fix(api): stop the automation sweep and retention jobs from stacking

Merged
SA
2mo ago
+116-17

Production has been hitting Neon's connection limits (Sentry API-22, API-24, API-2B; the 2 Sept Sazabi alert). Sampled on 2 Sept: 24 copies of the automation_events redispatch sweep running at once at up to 8 minutes each, 6 of prune-payloads, 4 of enforce-retention, together pinning ~34 backends for minutes, so ingest bursts then trip max_client_conn.

Two causes, both nominal to fix:

  • The sweep walks 1.2M rows it can never act on. redispatchUndispatched filters on dispatch_input IS NOT NULL, but automation_events_undispatched_idx only covers dispatched_at IS NULL. Rows recorded before #6635 with nothing to dispatch were never marked, so ~1.2M of them (planner estimate) sit at the index's left edge and every run walks them cold before reaching anything actionable. The sweep now looks back 24 hours, which turns the scan into a range that starts after them (EXPLAIN on production: index range scan, ~42k rows estimated) and doubles as a retry ceiling: a run fired for a day-old event is worse than no run.
  • The jobs stack. The three QStash-scheduled routes fire on cadences shorter than one batch takes, so every tick joins the previous ones. singleFlight takes a job-scoped pg_try_advisory_xact_lock and skips the tick when another invocation already holds it. The lock is transaction-scoped, so a function the platform kills mid-batch releases it on rollback, and batches run on the lock's own transaction so they hold one connection rather than two.

Also gives prune-payloads the same maxDuration = 300 as enforce-retention: a batch measured 224 s in production, so on the platform default the route was killed before its first batch returned.

Deliberately not an index change. Rebuilding the partial index on a 101 GB table needs a lock the ingest path cannot afford right now; the bound gets the same scan without it. Stuck rows older than 24 hours are abandoned by the sweep; production currently has no GitHub or Linear event triggers, so nothing user-visible depends on them.

Verified with tsc --noEmit and bun test src in apps/api; biome clean.

Summary by cubic

Prevents scheduled automation redispatch, payload pruning, and retention jobs from stacking by making overlapping invocations skip, reducing Neon connection pressure. The redispatch sweep now retries only actionable events from the previous 24 hours after a one-minute grace period, so older stuck rows are abandoned.

  • Transaction-scoped advisory locks protect each job batch, and cleanup responses report skipped and more when a later tick must continue.
  • prune-payloads now has a 300-second limit, matching enforce-retention.
  • The agent setup test removes state at the path the reader resolves, preventing leaked state from sibling test hooks.

Summary by CodeRabbit

  • Bug Fixes
    • Prevented overlapping automation evaluations, payload pruning, and retention runs from processing the same work concurrently.
    • Improved handling and reporting when a scheduled maintenance run is skipped because another run is already in progress.
    • Extended the payload-pruning time limit to allow larger cleanup batches to complete.
    • Limited redispatch checks to recent events, reducing unnecessary processing of older records.

Also carries a one-line hardening of packages/agent-setup/src/disabled-agent-hooks.test.ts, which failed 3 of 4 CI runs on this PR and passes locally: the "missing file" case now removes the state file at the path the reader resolves, since a sibling file's top-level hook can leave SUPERSET_HOME_DIR pointing elsewhere in CI and the previous test's write then survives afterEach.