fix(web): make webhook Retry deliver a call that used up its attempts
What was broken
Pressing Retry on a webhook call that had used up its 6 attempts did nothing. The call showed as pending forever.
Webhook jobs use the call id as the BullMQ job id, and failed jobs are kept for 30 days (DEFAULT_QUEUE_OPTIONS). BullMQ ignores add() for an id that already exists, so retryCall reset the row to PENDING but queued nothing. Reproduced against real Redis with BullMQ 5.79.3 (the version in the lockfile).
The fix
A small helper, requeueJob (apps/web/src/server/queue/requeue-job.ts), used only by WebhookService.retryCall:
- Kept failed job:
job.retry("failed", { resetAttemptsMade, resetAttemptsStarted }). One atomic BullMQ script; the job keeps its 6 attempts and backoff. - Job still waiting, delayed or running: left alone; it will deliver the call.
- No job: a new one is added.
New calls (emit, testWebhook) keep the plain add(), so the hot path does no extra Redis work.
Why not #516
This supersedes #516 (Polylane), which called queue.remove(callId) before every add(). For a job id that doesn't exist (every new call), BullMQ's remove script runs LREM over the wait, paused and active lists inside Redis: extra blocking work on the busiest path, growing with the backlog. It also deleted delayed and waiting jobs before re-adding them.
Tests
requeue-job.integration.test.ts: runs against real Redis in CI. It covers a plainadd()doing nothing after a failure,requeueJobre-running the job with its attempts reset, and jobs that are delayed or mid-run being left alone (no double run). With a plainadd()in place of the fix, the key test fails.webhook-service.unit.test.ts: Retry uses the new path, and new calls never look up existing jobs.- Locally: all unit (353), tRPC (23) and API (50) tests pass; the integration file passed 5 runs in a row; the production build (
build:editor+build:web) passes. - Reviewed twice by Astra. The second review found no blockers; its test-reliability notes are applied.
No migrations, no schema or config changes.
Ask Polylane for an impact assessment of this pull request:
Summary by cubic
Fixes webhook Retry so a call that used up its 6 attempts is actually delivered again, instead of being reset to pending and queuing nothing.
- Retry now goes through
requeueJob, which moves the kept failed job back to the queue with its attempts reset in one atomic step viajob.retry. - Jobs still waiting, delayed, or running are left alone; a new job is only added if none exists.
- If
requeueJobthrows, the call row is restored to its previous state, undoing only Retry's own reset so a worker's result overrides it. - New calls keep the plain
add(), so the hot path does no extra Redis work.
Summary by CodeRabbit
- Bug Fixes
- Retrying a webhook call avoids adding duplicate queued work. Failed or completed calls can be retried with their attempt count reset, while calls already queued remain in place.
- Calls missing from the queue are added for processing. Calls that do not belong to the specified team are rejected before queue access.
- If requeuing fails, the call’s previous status, attempt count, retry time, and response details are restored before the error is reported.