#6745 chore(api): bound both event logs to a 7-day payload window

The database is 580 GB. Raw provider webhook bodies are 459 GB of that, across two tables, neither of them bounded.

table	size	rows	pruner
ingest.webhook_events	532 GB	102.1M	written, never scheduled
public.automation_events	35 GB	4.1M	none, ever
everything else	13 GB

automation_events keeps its own copy on purpose — "ingest is prunable and the prompt needs this at dispatch" — which is correct, and also means once dispatch has happened that copy is as disposable as the first.

Why 7 days

Measured on production data: ingest runs at ~1.16M events/day, payloads average ~5.2 KB (Linear 3.3 KB across 69% of volume, GitHub 9.9 KB across the rest). That's ~4.8 GB/day, so the retention window is what sets the steady-state size.

window	payload storage
3 days	~17 GB
7 days	~34 GB
14 days (the old, unscheduled value)	~75 GB

Nulling, not deleting

automation_events.payload becomes nullable so it prunes the same way ingest does. The rows have to outlive their bodies: the unique index on (integration_connection_id, provider, external_event_id) is what makes delivery idempotent, so deleting a row means a redelivery of that event reprocesses instead of deduping.

Migration 0088 is a single ALTER COLUMN ... DROP NOT NULL. Generated, not applied.

Two carve-outs

google_calendar is excluded outright. Its ingest path (syncCalendar.ts:196) reads the most recent payload for a resource back to diff a calendar event against its previous state — and that row can be arbitrarily old. An event nobody has touched in months is exactly the one whose previous state still matters. It's 2,251 rows and 2.5 MB, so keeping all of it costs nothing worth measuring.

Rows still awaiting dispatch are left whole. There are none today — all 1,213,096 undispatched rows have a null dispatch_input, meaning nothing matched rather than nothing ran, so redispatchUndispatched has no backlog. But the sweep re-dispatches from dispatch_input and there's no reason for a pruner to race it.

After merge

Needs a QStash schedule for /api/automations/jobs/prune-payloads (the ingest one is already scheduled at */5 * * * *). I'll add it once this deploys, so it isn't hitting a 404 in the meantime.

Neither pruner reclaims disk on its own — nulling a TOASTed value leaves dead tuples that Postgres won't return to the OS without a rewrite. The 459 GB backlog gets cleared by the dump/restore in the us-east-1 region move; this PR is what stops it growing back.

Testing

• Lint, typecheck, and the api/db/trpc suites pass
• Two local failures are pre-existing and unrelated: @superset/mobile is missing react-native-reorderable-list from this worktree's node_modules, and pty-crash-ports-patch.test.ts (landed 2h ago in #6738) asserts a patches/node-pty patch this worktree installed before it existed. Both resolve on a fresh bun install --frozen in CI.

https://claude.ai/code/session_014D6wif8HDRCXEFf5Ku4w7V

---

Summary by cubic

Bounds raw webhook payload retention to 7 days for ingest and automations to stop unbounded growth while preserving deduplication. Previously, ingest targeted 14 days and automations kept payloads forever; now both null payloads older than 7 days, keep rows for idempotency, and exclude google_calendar.

• Adds /api/automations/jobs/prune-payloads: nulls automation_events.payload in 5k-row batches with time/row caps, skips google_calendar, and leaves rows awaiting dispatch intact.
• Improves pruner safety and speed: partial index on automation_events(received_at) WHERE payload IS NOT NULL; uses FOR UPDATE SKIP LOCKED and restates payload IS NOT NULL on UPDATE to avoid overlap and double-clears; cursor-based oldest-first scan; JSON response includes pruned, batches, and more.
• Changes ingest retention from 14 to 7 days; behavior otherwise unchanged. Notes 5–6 GB/day payload growth; retention window sets the steady-state size.
• Makes automation_events.payload nullable and adds the partial index via migration to keep dedupe rows while pruning bodies.

Rollout

• Run DB migration 0088_automation_events_payload_nullable.
• Schedule QStash for /api/automations/jobs/prune-payloads (ingest pruner already scheduled).
• Do not expect immediate disk reclaim; nulling TOASTed values frees space on rewrite. The upcoming dump/restore clears backlog; this PR prevents it from returning.

Summary by CodeRabbit

• New Features

  • Added incremental cleanup for older automation event payloads.
  • Cleanup reports progress and indicates when additional processing remains.
  • Important calendar and pending-dispatch payloads are preserved.

• Bug Fixes

  • Reduced ingest payload retention from 14 days to 7 days.
  • Payloads can now be cleared while retaining the associated event record.

• Data Management

  • Updated storage behavior to support more efficient long-term database maintenance.