fix(db,api): replace the 299 GB automation_events table and keep two days of it
Stacked on #7796 (remove Google Calendar). Merge that first; this PR's base then moves to main.
Summary
- Migration
0125swapsautomation_events(67M rows, 299 GB, +10 GB/day, no row retention) for an empty table of the same shape, carrying the last hour of rows, and drops the old one. Same shape as0122(#7792) foringest.webhook_events. automation_eventsjoins the 48-hour retention job.prune-payloadsand its partial index are deleted: at 48 h nothing ever reaches its 7-day window.- Database goes from ~366 GB to ~69 GB.
Why / Context
The pruner nulled bodies after seven days, which returns no space, and nothing ever deleted a row. Retention could not be added before because it needed two index builds under write locks on live tables (received_at here, event_id on automation_runs for the FK's SET NULL); on an empty table they cost nothing, which is what the swap unlocks. #7796 is what made the copy simple: the calendar sync was the only reader of rows older than an hour.
How It Works
packages/db/drizzle/0125_automation_events_swap_to_short_retention.sql, one transaction:
- Create
automation_events_swapwith identical columns/constraints, all current indexes minusprunable_idx, plusreceived_at_idx; createautomation_runs_event_idx. - Wait on the redispatch sweep's advisory lock (the
singleFlightkey), withSET LOCALtimeouts raised for that one wait and restored after. The sweep holds a transaction on this table up to a minute at a time; waiting on its lock lets the batch finish without queueing inserts behind an exclusive lock, and holding it makes the next batch skip. - Without a table lock: copy the last 60 minutes (bodies kept only for the last 15 — what a run still being dispatched reads; nothing reads a body after dispatch).
now()is the transaction start, so the bounds don't move. LOCK TABLE→ delta copy of rows withreceived_at > now()(arrived during step 3; index-only, tiny) →DROP TABLE … CASCADE(drops the runs FK) → rename table, indexes, unique, FK and not-null constraints into place.- Null the ~1,650
automation_runs.event_idvalues whose rows were not carried (whatON DELETE SET NULLwould have done) and re-add the FK.
Idempotency: Linear/GitHub/Notion/Slack redeliveries are deduped a layer earlier (ingest.webhook_events, 48 h). Gmail's history re-walk and Sentry/Teams retries land within minutes, inside the carried hour. Run-level dedup (automation_runs (trigger_id, event_id)) is untouched.
DB migration safety checklist (.agents/skills/db-migrations)
- Sizes checked on production 2026-09-24:
automation_events299 GB / 67M rows;automation_runs101k rows, 1,650 event refs - No column type change
- Lock order child-first:
automation_eventsis taken before theDROPtouchesauth.organizations(the FK parent) — the same order an insert takes them;automation_runsis locked last, for the FK re-add - ACCESS EXCLUSIVE on a hot table only for the delta + drop + renames + FK: ~1.3 s measured
- Every statement inside 60 s at production size: the hour's copy is the slowest (~25 s cold on a fresh branch, 3–6 s warm); the advisory wait deliberately raises
statement_timeoutto 11 min for itself and is not retried by the deploy (it retries only deadlock / lock-timeout) — a timeout there fails cleanly with nothing changed - Rehearsed 3× under pgbench on fresh production branches through
drizzle-kit migratewithPGOPTIONS="-c lock_timeout=5s -c statement_timeout=60s"(8 clients: inserts + the resource readback +dispatched_atupdates, with a simulated sweep batch holding its lock):
| pass | txns @ tps | errors | worst txn | throughput during the copy |
|---|---|---|---|---|
| 1 | 21,293 @ 178 | 0 | 495 ms | ~-8% for 10 s |
| 2 | 19,189 @ 160 | 0 | 1,232 ms | ~-30% for ~40 s |
| 3 | see comment below | 0 |
End state each time: table ~250 MB, 6 indexes, runs FK re-added, no _swap leftovers, branch 366 GB → 69 GB.
-
drizzle-kit checkpasses; a freshgeneratereports no diff - Hand-edited generated file, with Satya's explicit confirmation: drizzle emitted
CREATE INDEX … ON automation_eventson the live 299 GB table (the 17 September shape). The body was replaced by the swap, which creates the same indexes on the empty table; the snapshot is unchanged and correct. - Preview Neon branch: if this migration changes after CI first runs it, delete the
automation-events-swapNeon branch so the check re-runs for real.
Testing
bunx turbo typecheck(api, db, trpc);bunx biome check;bun testfor ingest, automations, ingest lib- Not app-verified: the retention job's first pass over
automation_eventsin production — it runs on the existing QStash schedule after deploy; the query is the same batch that already runs againstwebhook_events.
Rollout / Rollback
- Migrations run before the API deploys; the old code's retention loop ignores the new target until the API lands minutes later. The QStash schedule for
prune-payloadsshould be deleted in the Upstash console after deploy (console-only). - Rollback: revert the code normally. Dropped rows are recoverable only from a Neon branch created before the migration (PITR window 6 h); nothing in the codebase reads them.
Summary by cubic
Replaces the 299 GB automation_events table with an empty one that keeps only the last hour of rows (bodies for the last 15 minutes), adds 48-hour row retention, and drops the old table. This shrinks the database from ~366 GB to ~69 GB and stops the ~10 GB/day growth.
Changes
- Migration
0125swaps the table in one transaction: it carries over the last 60 minutes, waits on the redispatch sweep's advisory lock (raising its own timeouts for just that wait), and takes the table lock only for the delta copy and renames. automation_eventsjoins the 48-hour retention job;prune-payloadsand its partial index are deleted.- Adds
received_atonautomation_eventsandevent_idonautomation_runsso retention deletes and the FK'sSET NULLdon't scan. - Rows older than the carried hour are dropped permanently; redeliveries in that window still conflict on the dedup unique, and Linear/GitHub/Notion/Slack are deduped earlier in
ingest.webhook_events. - The migration is hand-written because drizzle's generated version built indexes on the live 299 GB table, repeating the September 17 outage; the snapshot is unchanged and correct.
- Stacked on the Google Calendar removal, the only reader of rows older than an hour.
Rollout
- Delete the
prune-payloadsQStash schedule in the Upstash console after deploy.
Summary by CodeRabbit
-
Bug Fixes
- Automation event records are now retained alongside webhook events, with older records and payloads cleared according to age-based retention.
- Automation runs remain available when their associated event records are removed; the event reference is cleared.
-
Documentation
- Updated database migration guidance to reflect the new retention behavior and lock-wait details.