fix(web): start the webhook cleanup job, deleting old finished calls in small batches (cloud)
What was broken
The nightly webhook cleanup job (webhook-cleanup-job.ts) has existed since webhooks shipped in January, but nothing imported it, so it never ran. Production logs across 26 deploys show no run at all. Webhook call history, with full payloads and responses, has been growing ever since.
The fix
- Cloud only. The job starts on cloud. Self-hosted installs keep their history until there's a setting to opt in, so nobody loses data on upgrade without being asked.
- Only finished calls are deleted. That means
DELIVERED,FAILEDorDISCARDEDcalls that were created and last updated over 30 days ago. TheDELETErepeats these conditions, so a call that someone retried after it was selected is kept. Pending and in-progress calls are never touched. - Small batches. It deletes 1,000 ids per batch. Each batch is its own transaction, with
lock_timeout 5s,statement_timeout 60sand a Prisma transaction limit set above them. There's a short pause between batches. A run stops starting new batches after 10 minutes, and the next night continues. A failed batch stops the run, and what was already deleted stays deleted. - Progress logs. It logs progress every 50 batches, plus a summary with elapsed time.
Why batches: the first run covers about 10 months of rows, on a database whose disk is about 78% full. One unbounded DELETE would be one long transaction with a burst of WAL. This doesn't shrink the database files: deleted rows become reusable space after autovacuum. What it does is stop the table growing.
Tests
webhook-cleanup-job.integration.test.ts: runs against real Postgres in CI. Only old, finished, untouched calls are deleted. Old pending, in-progress, recently retried and recent calls are all kept.webhook-cleanup-job.unit.test.ts: batching, the conditions repeated in the delete, the per-batch timeouts and transaction options, the deadline, and failure handling.instrumentation.unit.test.ts(from #507): the job starts on cloud with Redis, and doesn't start on self-hosted or without Redis.- Locally: unit (376), tRPC (23) and API (50) tests pass, and the production build passes.
- Reviewed by Astra over three rounds, with every finding applied.
No migrations, no schema changes.
Supersedes #507 (Polylane), which imported the job as-is: one unbounded DELETE of all calls older than 30 days, on every install.
Ask Polylane for an impact assessment of this pull request:
Summary by cubic
Makes the nightly webhook cleanup job actually run so webhook call history stops growing without bound. The job existed but was never imported, so it never started on any deploy.
- Bug Fixes
- Starts the job on cloud only; self-hosted installs keep their history until an opt-in setting exists.
- Deletes only
DELIVERED,FAILED, orDISCARDEDcalls created and last updated over 30 days ago, in batches of 1,000 with per-batch lock, statement, and Prisma transaction timeouts. - A run stops after 10 minutes and the next nightly run continues; a failed batch stops the run and keeps what was already deleted.
- New integration tests run the cleanup against a real Postgres in CI, and unit tests cover batching, deadlines, and failure handling.
Summary by CodeRabbit
- New Features
- Cloud installations using Redis now automatically clean up webhook calls that are more than 30 days old and have reached a final status: delivered, failed, or discarded. Cleanup runs in batches, and calls that are pending, in progress, or updated within the last 30 days are preserved.
- Self-hosted installations do not run this cleanup.