feat(core): opt-in DB_CREATE_IF_NOT_EXISTS and LOG_HTTP_REQUESTS

Merged
EV
15d ago
+3331-7

Summary

Two opt-in developer/operator conveniences in @gauzy/core. Both are off by default, so existing installs see no change:

  • DB_CREATE_IF_NOT_EXISTS=true: before any ORM connects, the API creates the configured PostgreSQL / MySQL database (DB_NAME) if the server doesn't have it yet. A fresh database server no longer needs a manual CREATE DATABASE.
  • LOG_HTTP_REQUESTS=true: logs one line per API request with method, path, status, duration, response size and request id. Secrets are never logged.

Why

  • #1769: TypeORM, MikroORM and Knex all need the database to exist already. If you point the API at an empty PostgreSQL server, it retries and then fails with database "gauzy" does not exist. Part 1 of that issue (SQLite as the default DB) was already done. This PR covers part 2: detect the configured DB type and create the database before the ORM starts.
  • #4006: developers, new ones especially, asked for a log line per request with endpoint, status code and execution time. The API had no general request log: ApiCallLogMiddleware only records time-tracking writes to the DB, and #4007 was closed without being merged.

What changed

packages/core/src/lib/database/provision/ (new)

  • database-provisioner.ts defines the DatabaseProvisioner interface, the provisioning target, and DatabaseProvisionError (which records the stage that failed: config, connect, check or create).
  • postgres-database.provisioner.ts:
    • connects to a maintenance database (DB_CREATE_MAINTENANCE_DB, default postgres) using the API's own host, credentials, TLS and connect timeout;
    • runs SELECT 1 FROM pg_database WHERE datname = $1;
    • only when the database is missing, runs CREATE DATABASE "<quoted name>";
    • treats 42P04 (another replica created it first) as "exists". After any other failed CREATE it looks the database up again: if another process created it meanwhile (23505, or 42501 for a role that couldn't have), that is "exists" as well;
    • runs the session with the API's own DB_STATEMENT_TIMEOUT / DB_LOCK_TIMEOUT and TCP keep-alive, and listens for the client's error event so a dropped connection can't crash startup.
  • mysql-database.provisioner.ts:
    • connects with no database selected;
    • runs SELECT … FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME = ?;
    • only when missing, runs CREATE DATABASE IF NOT EXISTS `<quoted name>` CHARACTER SET utf8mb4, with the same second lookup after a failure and the same error listener.
    • The lookup comes first so that a user with every privilege on an existing database except CREATE is not refused.
  • database-identifier.ts quotes identifiers (the quote character doubled inside the name; CREATE DATABASE can't take a bind parameter). It also validates the name: not empty, no NUL byte, at most 63 bytes on PostgreSQL and 64 characters on MySQL.
  • ensure-database-exists.ts is the flag, the per-DB_TYPE registry and the policy:
    • flag off: nothing connects and nothing is logged;
    • SQLite: no-op, since the driver creates the file itself;
    • database exists: one short connection and one lookup;
    • database missing: it is created. Nothing is ever dropped or altered;
    • missing and can't be created (no CREATEDB / CREATE privilege, or an invalid DB_NAME): startup stops with a clear message. The message names the privilege, the ALTER ROLE … CREATEDB / GRANT or CREATE DATABASE statement to run, and says you can unset the flag instead;
    • server unreachable, lookup failed, or no answer within the connect timeout + 60 s: one warning, then startup continues exactly as it would without the flag, so the ORM still reports and retries the connection. A role that can't connect to the maintenance DB but can connect to its own DB still starts.
  • Drivers (pg, mysql2, both already dependencies of core) are loaded lazily and injected, so the specs need no server. Supporting another engine means writing one more provisioner class.
  • A connection configured with a url option is read the way TypeORM reads it (its host, port, user, password and database win over the separate fields). On PostgreSQL the provisioner connects with that URL pointed at the maintenance database, so sslmode and certificate settings apply as they do to the API's pool.
  • At the deadline the provisioner is told to stop and closes its connection at once (pg drops the socket of a hung query; mysql2 is destroyed), so no socket is left to keep a seed or migration process alive.
  • Wired in bootstrap/index.ts → preBootstrapApplicationConfig, after plugin configurations, using the final dbConnectionOptions. Every entry point goes through that function: the API, yarn seed*, the migration commands and the worker.
  • Passwords are never logged. Messages carry only the driver's message, the host:port and the names. The driver error object is deliberately not attached as cause.

packages/core/src/lib/logger/http-request-logger/ (new, exported from the logger barrel)

  • HttpRequestLoggerModule.forRoot(overrides?) is a small Nest module that any Nest app can import. Its options are resolved when the application is created.
    • When disabled, configure() registers no middleware at all.
    • When enabled, it applies HttpRequestLoggerMiddleware to every route ({*splat}, i.e. everything under the /api global prefix) and logs one startup line.
  • HttpRequestLoggerMiddleware writes one line per request when it finishes, for example GET /api/employee?page=2 200 12.3ms 1.5kB [<x-correlation-id>].
    • A client abort is logged as … aborted after Nms.
    • 5xx responses are logged as warnings; everything else at log level.
    • It logs through the Nest Logger, so it follows LOG_LEVEL and whichever logger is installed. With Sentry's default SENTRY_LOG_LEVELS=error, these lines are breadcrumbs, not events.
    • Health checks (/api/health*) are skipped.
    • It never reads headers or bodies.
  • redact-request-url.ts:
    • redacts values of sensitive query parameters, matched by name words: token, password, secret, auth, key, session, cookie, signature, jwt, otp, code, pin, hash, email, state, verifier, nonce, ticket, handoff, …, including camelCase and bracketed names like where[accessToken] (so OAuth code / state / oauth_verifier callbacks and Zitadel ticket links are covered);
    • redacts a JSON query value (?data={"findInput":{"email":…}}) when any field name in it is sensitive or any string in it is a JWT or an email address. JSON that doesn't parse is redacted; a harmless filter is kept as it was;
    • checks a query value that is itself a URL (?redirect=/…, https://…, fragment included) with the same rules, and redacts it whole if anything in it would be redacted. Strings inside JSON values get the same checks, with one depth budget shared by JSON nesting and nested URLs;
    • redacts path segments that hold a sensitive route parameter. The router's req.params is read once the response is done, so /public/invoice/:id/:token is logged as /public/invoice/42/[REDACTED];
    • redacts anything that is or holds a JWT or an email address (alone, in text or in a list), wherever it appears;
    • writes the URL still percent-encoded, so a %0A can't split a log line;
    • caps the URL at 1024 characters.
    • It also redacts the segment that follows a /token/, /secret/ (and similar) segment, and a long random-looking segment (24+ token characters with letters and digits, not a UUID) that no matched route parameter accounts for. That covers a URL written before the request is routed, a 404 and a request the client abandoned, when route parameter names aren't known.
  • bootstrap/bootstrap.module.ts imports HttpRequestLoggerModule.forRoot().

Request-context debug line (core/context/request-context.middleware.ts)

  • Reuses the URL redaction for the request-context debug line. The existing "started" / "completed" lifecycle lines now write the URL through redactRequestUrl, the same redaction the request log uses. The "completed" line also knows the matched route parameters. Otherwise the line format is unchanged.

Docs

  • .env.sample: documents DB_CREATE_IF_NOT_EXISTS, DB_CREATE_MAINTENANCE_DB and LOG_HTTP_REQUESTS. All are commented out and default to off.
  • .env.local (the local-development template):
    • DB_CREATE_IF_NOT_EXISTS=false;
    • LOG_HTTP_REQUESTS=true, so local runs show request lines (see Decisions).
  • README.md:
    • one sentence in "Optional / Recommended for Production";
    • one note in the local-run notes, which also says the log covers /api and not /graphql.

Flags / settings and their defaults

VariableDefaultEffect
DB_CREATE_IF_NOT_EXISTSfalsetrue / 1: create DB_NAME on PostgreSQL / MySQL when it is missing. No effect on SQLite.
DB_CREATE_MAINTENANCE_DBpostgresPostgreSQL only: the database DB_USER connects to in order to create DB_NAME (for example template1).
LOG_HTTP_REQUESTSfalse (true in .env.local only)true / 1: one log line per API request. .env.sample, Docker / compose and production stay off. The production image does not ship .env.local.

Tests

Node 26, from the repo root. I ran only targeted suites because the machine is shared:

NODE_OPTIONS=--experimental-vm-modules npx jest -c packages/core/jest.config.ts --rootDir packages/core \
  src/lib/database/provision src/lib/logger/http-request-logger \
  src/lib/core/context/request-context.middleware.spec.ts src/lib/bootstrap/corp-policy.spec.ts

Result on the current head: 10 suites, 188 tests passed (143 on the first head). One run had a single failure, in an assertion I had not updated for the new signal argument; after the fix that suite passes on its own (29/29). corp-policy.spec.ts imports bootstrap/index.ts, so it also smoke-tests that the whole bootstrap module graph still loads with the new imports.

New suites:

  • database/provision/database-identifier.spec.ts: quoting, including injection-shaped names, and name validation in bytes vs characters.
  • database/provision/postgres-database.provisioner.spec.ts:
    • which connection options are used;
    • exists → no CREATE; missing → exactly one quoted CREATE;
    • 42P04 → exists; 42501 → the CREATEDB message, without the password;
    • 23505 / 42501 while another process creates the database → exists, after a second lookup; a failing second lookup keeps the CREATE error;
    • the API's statement / lock timeouts and keep-alive reach the session; the error listener is registered before connect();
    • a url reaches pg pointed at the maintenance database, query string kept; an abort while a query hangs ends the client at once;
    • other failures, connect failures and the 3D000 hint;
    • the connection is always closed.
  • database/provision/mysql-database.provisioner.spec.ts: the same matrix for MySQL, including both access-denied codes, the second lookup, the error listener, and destroy() (not end()) on an abort while a query hangs.
  • database/provision/ensure-database-exists.spec.ts:
    • flag off → no connection and no log;
    • SQLite → no-op;
    • create/config failures stop startup with a clear message; connect/check failures and unexpected errors only warn;
    • a server that never answers ends as a warning at the deadline (connect timeout + 60 s by default); the provisioner's signal is aborted then (and only then), and nothing it does afterwards surfaces;
    • session limits and a url option are read from the connection options as TypeORM reads them;
    • one end-to-end run through the built-in PostgreSQL provisioner.
  • database/provision/database-provision.drivers.spec.ts: the real pg and mysql2 drivers against a closed port. They load and accept the options, and an unreachable server ends as unverified (a warning), not a crash or a hang.
  • logger/http-request-logger/redact-request-url.spec.ts: the name matcher (OAuth names included), query and route-param redaction, JSON query values, URL query values (redirects, OAuth fragments, nested), JWTs alone or inside a value, emails, the /token/ rule, long random-looking segments (and the ids, dates and words that are kept), malformed encoding, no decoding of %0A, and truncation.
  • logger/http-request-logger/http-request-logger.middleware.spec.ts:
    • exactly one line on finish; 5xx as a warning; aborted requests;
    • a request carrying Authorization, cookie and API-key headers and a password body writes none of them;
    • extra redaction words; health checks skipped; LOG_LEVEL respected;
    • a 404 and an abandoned request redact a random-looking token segment.
  • logger/http-request-logger/http-request-logger.module.spec.ts:
    • disabled → no middleware; options resolved at app creation;
    • a real Nest app via supertest: it logs method/path/status/duration/size/request id, redacts a real :token route param and ?token=, redacts a token sent with a method no route takes (404) while keeping an id a route matched, never writes the Authorization or Cookie header, logs 500s, skips health, and is silent when disabled.

Also: type-checked all new files plus the dbConnectionOptions call site with a tsconfig scoped to packages/core/tsconfig.spec.json, with no errors. eslint on the touched files: no errors or warnings. prettier on the new files.

Manual verification still needed

  • Live databases. No PostgreSQL or MySQL server (or running Docker) was available here, so I did not run the full flow end to end. To check it, on PostgreSQL:

    1. drop gauzy;
    2. start the API with DB_TYPE=postgres DB_CREATE_IF_NOT_EXISTS=true;
    3. confirm DB_CREATE_IF_NOT_EXISTS: created database "gauzy". and that migrations run;
    4. restart and confirm … exists.;
    5. repeat with a role that lacks CREATEDB and read the message.

    Same steps for MySQL / MariaDB.

  • Running API. I did not run yarn start:api with LOG_HTTP_REQUESTS=true. The module spec boots a real Nest + Express app, but not the full Gauzy API.

Known limitations

  • The request log is Nest module middleware, so a request that Express middleware registered before it ends (a body that fails to parse, a payload over the size limit) gets no line. Logging those would mean mounting the logger with app.use() in bootstrap, ahead of the body parsers. That can be a follow-up if wanted.
  • /graphql is not covered (see Decisions).

Risk and rollback

  • With both flags unset (the default), behaviour is unchanged:
    • ensureDatabaseExists returns before doing anything;
    • HttpRequestLoggerModule.configure registers nothing.
  • When DB_CREATE_IF_NOT_EXISTS is on, the only new failure mode is a database that is known to be missing and can't be created. Startup would fail anyway in that case; it now fails right away with a clear message instead of after the ORM's retries.
  • Rollback: unset the flags, or revert the PR. No migrations, no schema changes and no new dependencies.

Decisions

  1. No standalone package for now. The provisioning stays a self-contained folder in core (packages/core/src/lib/database/provision/): drivers are injected and nothing from core is imported except @gauzy/config / @gauzy/utils in the entry file. That keeps a later extraction easy.
  2. .env.local turns LOG_HTTP_REQUESTS=true on. It is the local-development template, and #4006 asks for exactly this. .env.sample, Docker / compose and every production default stay false.
  3. /graphql is not covered. That is accepted and stated in the README and .env.sample. A line there would only say POST /graphql.
  4. The request-context debug line reuses the URL redaction (see above).

Closes #1769
Closes #4006

🤖 Generated with Claude Code