fix(ingest): let the retention loop run more than one batch

Merged
SA
2mo ago
+23-4

Found by the 2026-08-24 schema audit (#6819). One file, no schema change, no migration.

The bug

TIME_BUDGET_MS was 20s. A batch measures 35s mean against production — so the while (Date.now() < deadline && ...) check after batch one always failed, and a run deleted 5,000 rows instead of the 50,000 MAX_ROWS_PER_TABLE implies.

From pg_stat_statements over the job's first 43.5 hours:

callsrows_deletedrows_per_callmean_msmax_s
1,2266,130,0005,00034,915582

rows_per_call is exactly BATCH_SIZE on all 1,226 calls — the loop has never once run out of work, and MAX_ROWS_PER_TABLE has never once been the binding constraint.

Net effect: ~3.4M rows/day of drain against ~1.7M/day of intake, not the "~14M rows/day, comfortably ahead of the ~1.2M/day" the comment claimed. That leaves a 65M-row over-retention backlog on ingest.webhook_events (62% of the table) draining on a ~38-day horizon, and going break-even if intake grows — which it is: a like-for-like hour went 35,162 rows on 08-23 to 110,060 on 08-24.

Also: no maxDuration

This was the only job route under apps/api without one, so it took the platform default — shorter than a single batch now costs, so runs were being killed mid-loop rather than ending on their own budget. 300 matches the other long-running job routes (refresh-mrr, sweep-schedules, renew-watches).

Set to 300 with a 240s budget, so the budget binds first and ~6 batches fit.

What this does not fix

Batches are slow because each is ORDER BY received_at LIMIT 5000 off the left edge of webhook_events_received_at_idx, and every prior delete leaves dead index entries in exactly that prefix for the next batch to walk. Autovacuum isn't clearing them at any useful rate — at 105M live rows the default scale_factor = 0.2 puts the trigger near 21M dead tuples and the table sits at 3.4M. This PR only stops the time budget from capping the loop at one batch; it doesn't address the underlying scan cost.

Worth knowing: at ~6 batches/run the job will be deleting near-continuously. FOR UPDATE SKIP LOCKED already makes overlapping runs safe, but it does mean any ACCESS EXCLUSIVE DDL on ingest.webhook_events will now almost certainly queue behind a batch — which is why the unused-index drop the audit also recommended is being deliberately held until the backlog clears.

Verification

bunx turbo typecheck --filter=@superset/api → exit 0. Biome check on the changed file → clean. Not exercised against production; the numbers above are read-only measurements taken during the audit.

Summary by cubic

Lets the ingest retention job run more than one batch per run by raising the time budget and setting an explicit maxDuration. Previously, runs always stopped after the first 5k-row batch due to a 20s budget and platform timeouts; now the loop can reach the intended 50k rows per table per run.

  • Increase TIME_BUDGET_MS from 20_000 to 240_000 so the loop can execute multiple batches and hit MAX_ROWS_PER_TABLE (50_000).
  • Add maxDuration = 300 to the route to avoid platform default kills and match other long-running jobs.
  • No schema changes; no migrations.

Operational impact

  • Expect ~6 batches per run; accelerates backlog drain noted in audit #6819.
  • Overlapping runs remain safe via FOR UPDATE SKIP LOCKED; ongoing deletes may delay ACCESS EXCLUSIVE DDL on ingest.webhook_events.
  • Does not change batch scan cost; dead index tuples will still accumulate until autovacuum clears them.

Summary by CodeRabbit

  • Improvements
    • Increased the retention job’s processing time budget to support larger batches.
    • Added a five-minute maximum execution window for the retention process.
    • Updated operational guidance to clarify batch cost and deadline behavior.