db audit 2026-08-24: baseline — 419 GB dead TOAST, retention job at 24% of designed rate

Closed
SA
2mo ago
+862-0

Report only. This PR adds db-audit-findings.md and touches nothing else. Nothing was changed in the database: every statement was a SELECT or a plain EXPLAIN, run with statement_timeout='60s' and default_transaction_read_only=on.

Whether to act on a finding is a human call made after reading. Two were made and are recorded in the report so next week's diff isn't confusing: §2 was fixed in #6842 (no schema change), and the §5 index drop was deliberately deferred because DROP INDEX needs ACCESS EXCLUSIVE and would queue behind a retention batch, stalling webhook inserts. Everything else stands as found.

This run establishes the baseline

db-audit-findings.md does not exist on main and has no git history, so there is no "Changes since last report" section and no week-over-week comparison. Every number is a first observation. Next week's run can diff against it, provided the stats window has not reset in between.

Stats window: 31 days 5 h (postmaster and pg_stat_statements both reset 2026-07-24 11:00 UTC). Longer than the ~10-day floor, so idx_scan = 0 findings are reported at full confidence — read them as "unused for 31 days", never "unused ever".

What's urgent

1. 419 GB of dead TOAST on ingest.webhook_events — 68% of the 617 GB database. Migration 0090 is a bare ALTER TABLE ... DROP COLUMN "payload", which is catalog-only and reclaims nothing. The dropped column is still in pg_attribute as ........pg.dropped.5......... Sampling the one remaining toastable column (error) puts live toastable data in that relation at ~42 MB against a 419 GB TOAST heap.

2. enforce-retention runs at ~24% of its designed rate. The job (landed cb6df622f, 43.5 h before this audit) works, but its own comment claims "~14M rows/day, comfortably ahead of the ~1.2M/day these tables take on". Measured from pg_stat_statements: a 5,000-row batch averages 35 s against a TIME_BUDGET_MS of 20,000. The deadline is checked before each batch, so every invocation runs exactly one batch and deletes exactly 5,000 rows — an invariant, not an average — and MAX_ROWS_PER_TABLE (50,000) can never be reached. The route also sets no maxDuration, the only job route under apps/api that doesn't. Real capacity is 3.38M rows/day, intake is 1.66M/day, net drain 1.72M/day. The 65.4M-row backlog (62% of the table, plain EXPLAIN estimate) clears in ~38 days — around 2026-10-01 — but only if intake stays flat. It isn't: a like-for-like hour went 35,162 rows on 08-23 to 110,060 on 08-24. At sustained ~3.4M/day intake, net drain reaches zero and the backlog never clears. Worst observed batch: 582 s, past any normal serverless timeout.

These two are the same problem: the TOAST frees itself as the backlog drains, which is why #1 is not a separate task.

3. automation_events has no row retention, a ~2026-09-14 deadline, and its mitigation is cancelled. 54 GB at nine days old, growing ~600k–1.3M rows/day. It's deliberately excluded from the retention job, whose comment defers to a us-east-1 restore that would "rebuild both tables before then with those indexes created on the way in, for free".

That restore isn't in the repo, so I checked Linear. SUPER-191 "Migrate Neon database from US West to US East" is Canceled, last updated 2026-05-07 — three and a half months before that comment was written. It's the only matching issue including archived ones, there's no successor under another name, and there's no tracking issue for the automation_events retention or either index build. The free rebuild is not coming.

I verified the two enabling indexes really are absent: automation_events has no plain received_at index (both existing ones are partial and exclude exactly the rows retention needs, since payload is already NULL by then), and automation_runs_event_dedup_idx has event_id as its second column, so it can't serve the FK's ON DELETE SET NULL. Nothing is broken yet — only ~653 rows are past 30 days today — but the window to build those indexes calmly is the next three weeks.

Also found

  • 36 foreign keys with no supporting index; automation_runs.event_id is load-bearing for #3.
  • ~1.2 GB of unused indexes, incl. webhook_events_provider_status_idx (1,082 MB, 12 scans in 31 days, and its only occurrence in the repo is its own declaration at packages/db/src/schema/ingest.ts:44).
  • task_statuses.type holds duplicate and triage — 15,748 rows, 4.0% — against a schema comment listing five permitted values. Linear WorkflowState types arriving through the sync with no CHECK or enum to reject them.
  • github_pull_requests.state documents "merged", which has never occurred (exact indexed count: 0). Cosmetic — GitHub's REST API only emits open/closed and the tRPC reader already validates correctly — but it invites a silent zero-row query.
  • Only 3 CHECK constraints exist in the entire database.

Verified healthy, carried forward as deliberate

Keeping ingest.webhook_events unpartitioned so the dedup index can stay on (provider, event_id) is correct and the reasoning at packages/db/src/schema/ingest.ts:12-20 holds up — that index shows 109,420,619 scans in 31 days. Partition maintenance is healthy (14 future partitions through 2026-09-07, webhook_payloads_default holds 0 rows — the exact condition 0089 says to alert on). The automation_events.payload 7-day pruner lands exactly on its boundary. Events and payloads are exactly 1:1. No sequences and no int4 PKs exist, so category 9 is clean.

The pages v1 tables (#6806) got a full first-pass review against all ten categories and came back clean.

Summary by cubic

Adds a baseline production Postgres audit at db-audit-findings.md and corrects retention-job analysis; this sets accurate reference numbers and clarifies immediate storage/retention risks. Report-only; behavior unchanged.

  • Clarifies §2: each invocation deletes exactly one 5k-row batch due to a 20 s time budget; inferred cadence ~2.1 min; the route lacked maxDuration; worst batch 582 s; effective capacity ~3.4M rows/day vs intake 1.6–3.4M/day.
  • Notes follow-ups outside this PR: §2 fixed in #6842 (TIME_BUDGET_MS raised, maxDuration added); §5 index drop explicitly deferred to avoid ACCESS EXCLUSIVE during the backlog.
  • Updates §3: the planned us-east-1 restore is not coming — Linear SUPER-191 is canceled — so automation_events still lacks row retention and enabling indexes.
  • Review notes: report-only doc addition; all queries were read-only with a 60 s timeout; no rollout or migrations.

Summary by CodeRabbit

  • Documentation
    • Added a production database schema audit report with baseline statistics, methodology, and execution limitations.
    • Documented findings involving data retention, indexes, constraints, foreign keys, enum and comment consistency, cascading deletion, and schema health.
    • Recorded recent audit updates, intentional design decisions, recommended follow-up actions, and deferred items.
    • Clarified retention processing metrics and the status of a previously planned restore.
    • No database changes or fixes were applied.