fix(web): start the webhook cleanup job, deleting old finished calls in small batches (cloud)

Merged
KM
4d ago
+488-12

What was broken

The nightly webhook cleanup job (webhook-cleanup-job.ts) has existed since webhooks shipped in January, but nothing imported it, so it never ran. Production logs across 26 deploys show no run at all. Webhook call history, with full payloads and responses, has been growing ever since.

The fix

  • Cloud only. The job starts on cloud. Self-hosted installs keep their history until there's a setting to opt in, so nobody loses data on upgrade without being asked.
  • Only finished calls are deleted. That means DELIVERED, FAILED or DISCARDED calls that were created and last updated over 30 days ago. The DELETE repeats these conditions, so a call that someone retried after it was selected is kept. Pending and in-progress calls are never touched.
  • Small batches. It deletes 1,000 ids per batch. Each batch is its own transaction, with lock_timeout 5s, statement_timeout 60s and a Prisma transaction limit set above them. There's a short pause between batches. A run stops starting new batches after 10 minutes, and the next night continues. A failed batch stops the run, and what was already deleted stays deleted.
  • Progress logs. It logs progress every 50 batches, plus a summary with elapsed time.

Why batches: the first run covers about 10 months of rows, on a database whose disk is about 78% full. One unbounded DELETE would be one long transaction with a burst of WAL. This doesn't shrink the database files: deleted rows become reusable space after autovacuum. What it does is stop the table growing.

Tests

  • webhook-cleanup-job.integration.test.ts: runs against real Postgres in CI. Only old, finished, untouched calls are deleted. Old pending, in-progress, recently retried and recent calls are all kept.
  • webhook-cleanup-job.unit.test.ts: batching, the conditions repeated in the delete, the per-batch timeouts and transaction options, the deadline, and failure handling.
  • instrumentation.unit.test.ts (from #507): the job starts on cloud with Redis, and doesn't start on self-hosted or without Redis.
  • Locally: unit (376), tRPC (23) and API (50) tests pass, and the production build passes.
  • Reviewed by Astra over three rounds, with every finding applied.

No migrations, no schema changes.

Supersedes #507 (Polylane), which imported the job as-is: one unbounded DELETE of all calls older than 30 days, on every install.

Ask Polylane for an impact assessment of this pull request:

Assess with Polylane


Summary by cubic

Makes the nightly webhook cleanup job actually run so webhook call history stops growing without bound. The job existed but was never imported, so it never started on any deploy.

  • Bug Fixes
    • Starts the job on cloud only; self-hosted installs keep their history until an opt-in setting exists.
    • Deletes only DELIVERED, FAILED, or DISCARDED calls created and last updated over 30 days ago, in batches of 1,000 with per-batch lock, statement, and Prisma transaction timeouts.
    • A run stops after 10 minutes and the next nightly run continues; a failed batch stops the run and keeps what was already deleted.
    • New integration tests run the cleanup against a real Postgres in CI, and unit tests cover batching, deadlines, and failure handling.

Summary by CodeRabbit

  • New Features
    • Cloud installations using Redis now automatically clean up webhook calls that are more than 30 days old and have reached a final status: delivered, failed, or discarded. Cleanup runs in batches, and calls that are pending, in progress, or updated within the last 30 days are preserved.
    • Self-hosted installations do not run this cleanup.