PR #10976 - ever-co/ever-gauzy

Why every API request is slow, and why all requests sometimes hang ~60 s

Measured on a local copy of the stack (PostgreSQL 17, Redis 8, NODE_ENV=production, Node 26.12, demo seed, image_asset/screenshot on the S3 provider so presigning runs as in production), develop vs this branch, same database and env. Harness: tools/perf/api-bench.mjs (sequential latency, 16 concurrent clients × 10 s, SQL statements and Redis commands per request from pg_stat_statements / INFO commandstats). Raw results and tables: tools/perf/results/.

The "~60 s" signature

The API deployments had no probes. ingress-nginx's default proxy_read_timeout is 60 s, after which it retries on the other pod. A pod whose event loop is blocked kept receiving traffic, so every request routed to it waited exactly 60 s, then "continued".
On develop the event-loop monitor added here (API_EVENT_LOOP_MONITOR_MS) showed the loop blocked up to 1.5 s under load while the pg pool never had a waiter: the stalls are CPU on the main thread, not the database.
What blocks or parks a pod: the Documents pipeline (PDF/XLSX/OCR, seconds of CPU per file) consumed inside API pods by default; the Redis session store and the Redis L2 cache awaited with no deadline (a half-open socket parks every request); an unbounded memory leak in the server-side Jitsu client (every entity write deep-copied into history forever; per-write sync cost 0.03 → 1.9 ms after 80k writes, measured with the real @jitsu/js, which is already the latest version); heap limit 12 GB in a 2 GiB container (OOM-kill instead of GC).

Per-request overhead (each fix is its own commit with the measurement in its message)

Area	Problem	Fix
Sessions	`saveUninitialized: true` stored a new session for every cookieless request (≈ all traffic): a Redis SET before `res.end`, or a heap leak with the memory store	`saveUninitialized: false` (`EXPRESS_SESSION_SAVE_UNINITIALIZED=true` restores), session-store deadline `REDIS_SESSION_TIMEOUT_MS`
Redis L2 cache	read awaited on every L1 miss, no timeout	`REDIS_CACHE_READ_TIMEOUT_MS` (250 ms) → miss
Entity load hooks	`setTimeout` per row (user/org/tenant/team/project/contact/…) and `new FileStorage()` per icon row, all before TypeORM releases the connection	promise published on the image asset; cached LOCAL provider
S3/Wasabi/DO	new `S3Client` per presigned URL (100 avatar rows = 454 ms blocked loop)	client cache per config + presigned URL reuse 10 min
TypeORM	~50% of API CPU in TypeORM, ~20% re-deriving 5 inherited `@RelationId`s per entity that are already columns	read-path skip + memoised getter (`TYPEORM_SKIP_REDUNDANT_RELATION_IDS=false` disables)
Tenant scoping	relation JOIN on every scoped read → TypeORM's 2-query DISTINCT path (`findOne` 2 statements, pages 3)	`EXISTS` predicate, same semantics incl. soft-deleted tenant/employee
JWT	user lookup 2 round trips; employee + membership sequential; passport `done` called twice → an exception thrown per request; DB outage answered 401 (refresh storms)	1 query; concurrent lookups; return/throw contract; 503 on DB unavailability
Validation	class-validator asked Nest's container per constraint → `UnknownElementException` thrown 10–40× per request	cached container adapter
`/tags`	25 many-to-many LEFT JOINs + COUNT(DISTINCT) + GROUP BY (cartesian product) — 1.2 s	correlated subqueries: ~60 ms, identical rows
Tenant settings	empty settings never cached → 2 queries per request	cached (60 s for empty)
Logging	`LOG_LEVEL` ignored (debug/verbose on), ~4.7 sync stdout lines per request from guards	`LOG_LEVEL` honoured; trace lines → debug
Pools	min 0 + 10 s idle → reconnect churn (TLS + SCRAM on the 4-thread libuv pool)	idle 30 s, `DB_POOL_MIN` 2, keepAlive, `application_name`, opt-in server timeouts, `UV_THREADPOOL_SIZE` 16
Boot	unused MikroORM connected eagerly and could fail boot; production build of develop did not boot (Terser minified the copied `@gauzy/*` packages → `InventoryPlugin_1`)	lazy MikroORM connect under TypeORM; `info.minimized` on copied packages
Deploy (prod + stage manifests)	no probes; heap > container; Docs processing in API pods	startup/readiness/liveness probes, `--max-old-space-size=1536`, `GAUZY_DOCS_QUEUE_WORKER_ENABLED=false` on the API (worker consumes)
HTTP	Node keepAliveTimeout 5 s < nginx upstream keepalive → sporadic 502	65 s (`API_KEEP_ALIVE_TIMEOUT_MS`)

Opt-in diagnostics added (all off by default): API_PERF_METRICS, API_SERVER_TIMING (Server-Timing header with SQL time/count and pool wait), API_SLOW_REQUEST_LOG_MS, API_EVENT_LOOP_MONITOR_MS (loop delay + pg pool waiting count).

Results

Same database, env and machine; develop (+ opt-in instrumentation only) vs this branch. 60 sequential requests, then 16 concurrent clients × 10 s. Two pairs are committed under tools/perf/results/, each with both legs measured back to back:

run 1 (*.run1.json, baseline-vs-fixed4.run1.md): machine otherwise idle — the table below.
run 2 (baseline.json / fixed4.json, baseline-vs-fixed4.md): recorded by the current harness with provenance (commits, invocation, hardware, raw samples, correctness summary) while unrelated desktop processes kept the CPU at 50–79%; absolute times on both legs are inflated by that, the direction is the same (1.6–12.6×).

Endpoint	seq p50 before → after	req/s @16 before → after	p99 @16 before → after
`GET /tags`	1218 → 66 ms (18×)	3 → 50	6430 → 774 ms
`GET /role-permissions/me` (164 KB)	136 → 28 ms (4.9×)	5.1 → 48	6331 → 799 ms
`GET /user/me`	56 → 11 ms (5.1×)	25 → 133	1533 → 478 ms
`GET /employee/pagination`	85 → 28 ms (3.1×)	12.5 → 61	1669 → 633 ms
`GET /organization-projects/pagination`	62 → 15 ms (4.2×)	31 → 104	747 → 342 ms
`GET /tasks/pagination`	39 → 13 ms (3.0×)	38 → 107	570 → 193 ms
`GET /organization-team/pagination`	45 → 14 ms (3.1×)	32 → 102	1121 → 211 ms
`GET /organization-team/me`	47 → 12 ms (3.9×)	41 → 136	561 → 144 ms
`GET /employee/:id`	35 → 16 ms (2.2×)	37 → 104	690 → 492 ms
`GET /health`	14 → 5 ms	130 → 384	519 → 67 ms
public, no DB	5.5 → 0.8 ms (6.8×)	353 → 1446	77 → 21 ms

Redis commands per authenticated request 1 → 0 (the session SET is gone); SQL statements per request −1 on every authenticated endpoint. tools/perf/response-diff.mjs compared every endpoint's JSON between the two builds (fixed4.response-diff.json): same rows everywhere; pages hold the same rows, within a page rows are now in primary-key order, which the old second query did not guarantee.

Second pass: everything that runs per request (commit `4024ab66c4`)

A CPU profile after the first pass showed what was left: class-validator tested every registered class with instanceof on every validate(), class-transformer re-derived expose/exclude metadata per object, TypeORM rebuilt its property-name replacement table and ran a regex over the whole SQL for every query and re-filtered ~105 subscribers per entity per load, Express 5 re-parsed req.query on every read, MikroORM forked an EntityManager for every request under TypeORM, and the Sentry middleware copied bearer tokens and cookies into the event scope. All of those depend only on boot-time metadata and are now memoised (each with an env switch back: CLASS_VALIDATOR_MEMOIZE_ANCESTORS, CLASS_TRANSFORMER_MEMOIZE_METADATA, TYPEORM_SKIP_REDUNDANT_RELATION_IDS), or no longer done.

Not changed (owner decision)

Production builds run with environment.production === false: apps/api/project.json fileReplacements points at packages/config/src/environments/environment.ts, which does not exist (the file is under src/lib/environments). Fixing it changes many defaults at once (THROTTLE_ENABLED, i18n watch, …).
packages/plugins/auth-keycloak has the same passport double-done pattern.
mutating-route-permissions.spec.ts fails on develop (EmployeeSettingController.patchData has no permission metadata, since 40f220e77c); it fails identically on this branch and is not touched here.

No feature, endpoint, field or option was removed; every behaviour change has an environment switch back to the old behaviour where one is meaningful.

🤖 Generated with Claude Code