feat(ingest): bound row growth on webhook_events
Closes the last unbounded thing in this area.
Bodies are already bounded — webhook bodies by dropping day partitions, automation payloads by the pruner. Rows never were. webhook_events sits at 105.7M and takes 1.16M/day; automation_events adds another 758K/day. Nothing deletes from either.
Why delete, not partition
The dedup indexes have to stay where they are. A unique index on a partitioned table must contain the partition key, and adding received_at to (provider, event_id) means a redelivery — which arrives with a different timestamp — no longer conflicts. Dedup silently stops working. That's the same trap #6749 avoided, and it applies here too.
Deleting is cheap now in a way it wasn't a week ago: with bodies moved out, these rows are ~350 bytes and touch no TOAST, so a batch leaves very little for autovacuum to chase. The original objection to DELETE was about churning 5 KB TOASTed values — that's gone.
Retention: 30 days
Set by idempotency, not storage. The unique indexes are what stop a redelivered event being processed twice, and a row that no longer exists can't dedupe. Providers redeliver within hours to a few days, so 30 is well past any real replay window. It's one constant if you want it tighter.
Measured on a copy of production
webhook_events, 5,000-row batch | 403ms |
| Plan | Index Scan → Tid Scan, no sort |
automation_events batch | 0 rows — correct, the table is 7 days old |
automation_events_received_at_idx build | ~20s on 4.5M rows |
| Backlog past 30 days | 65.2M rows |
| Paced drain (50k/table/run, every 5 min) | ~14.4M/day → backlog clears in ~4.5 days |
Notes
- The index build holds a write lock for ~20s on
automation_events. Brief, but it's on an ingest path — worth applying at a quiet moment, or asCONCURRENTLYby hand if you'd rather. - Only FK involved is
automation_runs.event_id → automation_events.id, which isON DELETE SET NULL. Checked; that's the documented intent. webhook_payloadsneeds no equivalent — its partitions drop at 7 days, so a body is always gone well before the event row that pointed at it.- Needs a QStash schedule for
/api/ingest/jobs/enforce-retentionat*/5 * * * *after deploy.
https://claude.ai/code/session_014D6wif8HDRCXEFf5Ku4w7V
Summary by cubic
Bounds row growth by enforcing 30‑day retention on ingest.webhook_events. Previously rows were never deleted; now a QStash‑triggered job deletes aged rows, setting the dedup window to 30 days without affecting bodies.
- Adds POST
/api/ingest/jobs/enforce-retention: deletes in 5k batches, up to 50k rows per table per run, ordered byreceived_atwithSKIP LOCKED, under a 20s per‑target budget. - Scope: only
ingest.webhook_eventsnow.automation_eventsis deferred to avoid locking index builds and will join after the restore seeds the needed indexes. - Required: create a QStash schedule for
/api/ingest/jobs/enforce-retentionat*/5 * * * *after deploy.
Summary by CodeRabbit
- Bug Fixes
- Retention processing now applies only to webhook events.
- Processing time is allocated independently for each configured target, improving consistency when handling multiple targets.