perf(ingest): split webhook bodies into a day-partitioned table

Merged
SA
3mo ago
+7289-61

Not for admin-merge. This changes the hottest write path in the product (44M+ inserts/week across three routes) and is coupled to the us-east-1 move — see Sequencing.

Why

ingest.webhook_events is 532 GB of a 580 GB database, 424 GB of it raw provider bodies in TOAST. Nulling payloads reclaims nothing: dead TOAST only returns to the OS on a rewrite. So retention-by-UPDATE (what #6745 does) caps growth but can never shrink the table. Partitioning makes retention a DROP, which returns space immediately.

The blocker, and why the split runs the other way

A unique index on a partitioned table must include the partition key. Adding received_at to (provider, event_id) silently disables dedup — a redelivery has a different timestamp, so it no longer conflicts. Verified on a scratch database:

CREATE UNIQUE INDEX ... ON (provider, event_id, received_at);
-- same event, delivered a day apart, ON CONFLICT DO NOTHING
-- rows_for_one_event: 2

So webhook_events stays unpartitioned and keeps its dedup index untouched. Only the bodies move to ingest.webhook_payloads, partitioned by day. Nothing reads a body back, so that table is write-only and isn't in the drizzle schema — it's created by a custom migration, since drizzle has no PARTITION BY syntax.

One round trip

recordWebhookDelivery writes both tables in a single statement, so the write path costs the same as before and the tables can't diverge.

Its final SELECT reads from the event CTE, not the payload insert. On a redelivery the body already exists, ON CONFLICT DO NOTHING returns zero rows, and returning from that insert would report a good delivery as failed. That was a real bug in my first version, caught in the scratch database.

Tested against a real copy of production

A Neon branch off production — 532 GB, 102.1M rows:

result
0089 DROP COLUMN payload on 102.1M rows103ms — catalog-only, no rewrite
0090 partitioned table + DEFAULT + maintenance fn~500ms
Maintenance: create 14 days ahead15 partitions, 172ms
Retention: drop a 50k-row partition35ms, space returned immediately
DELETE of the same volume, for comparison226ms, 46 MB still on disk

Then 500 real production payloads (GitHub + Linear) replayed through recordWebhookDelivery:

pass 1 (fresh):     500/500 recorded, 0 null
pass 2 (redeliver): 500/500 returned the SAME id, 0 null
events   +500 (not +1000)   payloads +500
round-trip fidelity: 200 identical, 0 mismatched
partition routing: 1000 rows in today's partition
                   webhook_payloads_default: 0 rows

Also confirmed jsonb still rejects NUL, so stripNullChars remains load-bearing — all three routes still call it.

Operational notes

  • A missing partition is a hard insert failure (no partition of relation found) — on this path that's 500s and eventual event loss once a provider stops retrying. Hence the DEFAULT partition. It must stay empty; alert on count(*) FROM ingest.webhook_payloads_default > 0.
  • Recovery when DEFAULT has rows is a drain-and-reattach in one transaction; verified working.
  • maintain_webhook_payload_partitions() needs a daily QStash schedule. Not created yet — it needs a job route, which I'd rather add in the same PR that wires the cutover.

Sequencing

The migration is sub-second even at 532 GB, so it's safe to apply anytime. But it drops the existing bodies — intentional, since retention discards them anyway, at the cost of losing the current debugging window at cutover.

The natural time to apply it is during the us-east-1 move: you restore the last 7 days straight into the partitioned table, so there's no gap and no separate backfill over a 500 GB table.

Testing

  • Lint and typecheck pass
  • Everything above ran against a real production copy, not a fixture
  • The replay harness is in my scratchpad rather than the repo; happy to add it as a script if that's useful

https://claude.ai/code/session_014D6wif8HDRCXEFf5Ku4w7V

Summary by cubic

Writes webhook bodies to a day-partitioned ingest.webhook_payloads while keeping ingest.webhook_events for identity/state; retention becomes a DROP that frees space immediately with dedup on (provider, event_id) unchanged. This is expand-only: the legacy payload column remains for existing rows; new deliveries write bodies only to the partitioned table.

  • recordWebhookDelivery writes event state and body in one statement and returns from the event CTE to preserve redelivery semantics; GitHub/Linear/Notion routes now call it.
  • ingest.webhook_payloads is write-only, partitioned by day with a DEFAULT partition, created via custom SQL and excluded from @superset/db schema; webhook_events stays unpartitioned and retains the unique (provider, event_id).

Rollout and operations

  • Apply migration 0089_webhook_payloads_partitioned (adds the partitioned table and maintenance function; no data drop).
  • Schedule the new job route /api/ingest/jobs/maintain-partitions via QStash to create 14 days ahead and drop partitions older than 7 days. Required: add the schedule.
  • Monitor count(*) FROM ingest.webhook_payloads_default; if > 0, drain and reattach those dates before creating their partitions.
  • No behavior change for event identity, dedup, or retries; the hot path remains a single round trip.

Summary by CodeRabbit

  • New Features

    • Improved webhook delivery tracking across GitHub, Linear, and Notion integrations.
    • Added reliable handling for duplicate and failed webhook deliveries, including retry tracking.
    • Added scalable webhook payload storage with automated partition maintenance and retention.

  • Bug Fixes

    • Prevented duplicate webhook payloads from being stored during redeliveries.
    • Improved consistency when recording webhook events and their processing status.