PR #7792 - superset-sh/superset

Summary

Migration 0122 swaps ingest.webhook_events (210M rows, 540 GB) for an empty table of the same shape and drops the old one, carrying over the last 15 minutes of rows.
Retention drops from 30 days to 48 hours; webhook payload partitions from 7 days to 1.
Database goes from ~1,069 GB to ~390 GB: ~537 GB at commit, ~140 GB more as old payload partitions drop over the following days.

Why / Context

Measured on production on 2026-09-19 (full write-up in memory db-load-snapshot-2026-09-19):

Webhook bookkeeping was ~68% of all DB time. The webhook_events upsert alone was 25%, at 25.6 ms mean and 6.9 s max — past Linear's 5 s webhook timeout.
The 30-day retention job could not keep up: each deleted row touches four indexes totalling 42 GB (the (provider, event_id) unique index alone is 28 GB) against a 6 GB cache, so a 5,000-row batch cost 126 s. The table gained ~3.6M rows a day net; the planner's histogram put the median row at 23 days old and the oldest at 90.
424 GB of the table is dead TOAST from the dropped payload column. No vacuum returns it; only dropping the relation does.

Draining the backlog by DELETE was never going to converge (~8 weeks at the measured rate while intake continues), so the table is replaced rather than emptied.

How It Works

packages/db/drizzle/0122_webhook_events_swap_to_short_retention.sql, in one transaction (drizzle's migrator):

CREATE TABLE ingest.webhook_events_swap with identical columns, defaults and indexes.
Wait on the retention job's advisory lock (hashtextextended('job:ingest.enforce-retention.webhook_events', 0), the key singleFlight derives). That job holds a DELETE transaction on this table almost continuously; an ACCESS EXCLUSIVE request queued behind it would lose the deploy's 5 s lock_timeout on every retry and stall every webhook insert while it waited. Waiting on the job's lock instead lets its batch finish without queueing anything, and holding it makes the job's next batch skip. SET LOCAL raises the timeouts to 11 min for this one wait and restores them to the deploy's values straight after (verified: 5s / 1min come back).
LOCK TABLE ... ACCESS EXCLUSIVE before the copy, so nothing lands in the table between the copy and the drop.
Copy the last 15 minutes (~20k rows, 192 ms). Deliveries still in flight through QStash look their row up by id; without this they would be skipped as "event missing".
DROP TABLE, rename the new table and its indexes into place, and rename the six not-null constraints so the result matches what a fresh CREATE TABLE from the schema builds.

Nothing references webhook_events by foreign key. Automation dedup lives on automation_events (integration_connection_id, provider, external_event_id) and is untouched.

Retention at 48 h covers every automatic redelivery a provider here attempts — Slack 5 min, Linear 1 min / 1 h / 6 h, Notion 8 tries over ~24 h — with a day to spare. Slack agent delivery is the one path that dedupes on this row alone; the others are caught a second time on automation_events. A manual GitHub redelivery older than two days reprocesses, which is what clicking redeliver asks for.

Payloads at 1 day: nothing reads a body after the 75-minute Linear abandoned-delivery sweep; this is the shortest a day partition allows. prune-payloads (automation_events bodies) is deliberately unchanged — it is an UPDATE on a 245 GB table, so pruning sooner adds churn and returns nothing.

DB migration safety checklist (`.agents/skills/db-migrations`)

Production sizes checked: ingest.webhook_events 540 GB / 210M rows (2026-09-23)
No column type change
Lock order: only webhook_events is locked; it has no FK parents and no FK children
ACCESS EXCLUSIVE on a hot table is held for the copy + drop + renames only: ~0.9 s measured at production size, including the 540 GB DROP
Every statement inside 60 s at production size — with one deliberate exception: the advisory-lock wait in step 2 raises statement_timeout to 11 min for itself. Retention batches now measure ~53 s (were up to 582 s on the 2 CU compute). A timeout there is not retried by the deploy (it retries only on deadlock detected / lock timeout); it would fail the deploy cleanly with nothing changed.
Rehearsed on Neon branches of production under load, 3 passes on fresh branches, through drizzle-kit migrate with PGOPTIONS="-c lock_timeout=5s -c statement_timeout=60s":

pass	pgbench (8 clients, upsert + read + status update, with a retention batch holding its lock)	worst txn	throughput
1	13,393 txns @ 150 tps	992 ms	dipped 153 → 132 tps for one 10 s window
2	12,783 txns @ 143 tps	1,338 ms	one 10 s window
3	13,235 txns @ 148 tps	1,014 ms	one 10 s window

Every bench row was present in the new table afterwards; the old table was gone; __drizzle_migrations recorded the migration.

Reads move to a new table and the data that matters (in-flight rows) moves in the same statement
Preview Neon branch: first push, nothing to delete. If the migration file changes after this push, delete the db-load-webhook-analysis Neon branch so the preview check runs for real.

Testing

bunx turbo typecheck --filter=@superset/api --filter=@superset/db
bunx biome check on the touched routes
bun test in apps/api for src/lib/ingest, src/app/api/ingest, src/app/api/automations, src/app/api/integrations/linear (43 pass)
bunx drizzle-kit check; a fresh drizzle-kit generate reports no schema diff
Verified empirically that PGOPTIONS reaches the server through drizzle-kit's driver (a 1ms statement timeout kills the migration), so the deploy's lock bounding is real.

Rollout

Migrations run before the API deploys, so for a few minutes the old code (30-day retention) runs against the new, nearly-empty table. That is harmless: there is nothing older than 30 days to delete.
The first maintain-partitions run after deploy drops ~5 payload partitions in one go instead of one.
Neon's synthetic storage reflects the drop once the 6-hour PITR window passes.

Rollback

Code: revert normally. The retention constants are independent of the migration.
Data: the dropped rows are recoverable only from a Neon branch created from before the migration ran (PITR window 6 h). Nothing in the codebase reads them; the value of a 2-day-old webhook event row is dedup of a redelivery, and every automatic redelivery lands well inside 48 h.

Follow-ups

automation_events is now the largest table: 245 GB, 55M rows, ~2.3M rows/day with no retention at all. It needs the same swap, but its automation_runs.event_id FK and the indexes that retention would need have to be built CONCURRENTLY, outside drizzle's transactional migrator. Separate change.
GitHub check_run / workflow_run / check_suite events are 69% of intake and rewrite the same ~880 PR rows ~27 times an hour. Coalescing those writes is the next-largest lever.
webhook_events_provider_status_idx was scanned 6 times in 9 days; it was kept for a like-for-like swap and can go separately.

Summary by cubic

Replaces the 540 GB, 210M-row ingest.webhook_events table with an empty one carrying over only the last 15 minutes of rows, cutting database size from ~1,069 GB to ~390 GB and retention from 30 days to 48 hours.

Migration

Creates an identical empty table, copies in-flight rows, drops the old one, and renames the new one into place without blocking webhook inserts.
The old table could not be drained by deletion: each 5,000-row batch cost 126 s against a 6 GB cache while intake added ~3.6M rows a day, and 424 GB of dead TOAST could only be reclaimed by dropping the table.

Retention

48 hours covers every automatic redelivery (Slack, Linear, Notion) with a day to spare; Slack agent delivery dedupes on this row alone.
Payload partitions drop from 7 days to 1 day; nothing reads a body after the 75-minute Linear sweep.

Summary by CodeRabbit

Maintenance
- Webhook event records are now retained for 48 hours instead of 30 days, and webhook payloads for one day instead of seven.
- As part of the update, existing webhook event history older than 15 minutes was cleared; recent events were preserved.
API Changes
- The retention-job response now reports the retention window in hours rather than days.