fix(jitsu-analytics): stop the entity-event flood that breaks every signup

Merged
EV
18d ago
+723-18

Problem (production)

Every signup has failed since the 2026-10-27 release; the last successful onboarding was 10-26 14:28Z. On 10-27 alone, 27 empty tenants were left behind. POST /tenant creates the tenant and its roles, then times out connecting to the database before it links the creator to the tenant. The creator still has no tenant, is sent back to onboarding, and each retry creates one more empty tenant.

Cause

JitsuEventsSubscriber is a global TypeORM subscriber. For every row the API writes, it starts an HTTPS track() and does not wait for it. Onboarding writes about 3,300 rows in roughly a second: 8 roles, 3,136 role permissions, 91 feature toggles, and so on. One API pod therefore opened thousands of TLS connections to data.gauzy.co at once. Prod logs show 2,300–3,400 data.gauzy.co/api/s/track failed … socket hang up / EPIPE / TLS disconnect per attempt.

The event loop stalled. Redis "did not answer within 250ms", and pg-pool connects hit their 5 s timeout (timeout exceeded when trying to connect). Feature-toggle seeding took 14–42 s, the parallel seeds failed, and the request died before the user-link step. Postgres was fine throughout (247/300 connections, 19 of them gauzy).

Fix

Entity events now go through a BoundedEventQueue:

  • Nothing is sent on the writer's stack; sending starts on the next tick.
  • At most JITSU_MAX_CONCURRENT_EVENTS (default 4) events are in flight.
  • At most JITSU_MAX_QUEUED_EVENTS (default 1000) wait for a slot.
  • Anything beyond that is dropped and counted, never copied or redacted. The count is logged at most once a minute, and the rest of a burst is reported when the interval ends.

This alone stops the failures. A user stuck by an earlier failure can simply retry: a fresh tenant is created and linked.

Also in this PR: four words develop's cspell run flags (ECONNREFUSED, nodemailer, topbar, Diaz).

Moved out of this PR

Resuming the empty tenant a failed attempt left behind (instead of creating another) is on fix/tenant-onboarding-resume. Review found that it needs a per-user lock (concurrent retries could duplicate roles/seeds), idempotent role and feature reconciliation (migrations add partial rows to empty tenants), and imageId handling. It will come as its own PR.

Tests

  • jitsu-event-queue.spec.ts, 20/20, stable over 3 runs:
    • a 3,136-event burst keeps 4 in flight;
    • every queued event is sent eventually;
    • drops happen without building the event and are reported once per interval, with the rest reported at the end of the interval (this test fails against the previous commit);
    • a failed send doesn't stall the queue;
    • env values are parsed strictly.
  • cspell clean.
  • Typecheck shows no errors in the changed files.

🤖 Generated with Claude Code