fix(web): re-dispatch a webhook call when it is retried
Retrying a failed webhook delivery returned success but sent nothing. The retry reused a job id that BullMQ still held for the failed attempt, so the new job was dropped and the call stayed pending. The retry now queues a fresh job, so the subscriber receives the event. BullMQ ignores add() when a job id already exists, and failed jobs are kept for 30 days, so the retry was dropped. The resolved BullMQ add script and the retained-job option show the collision; successful calls are unaffected because completed jobs are removed immediately.
Production impact at 3c7e57a: Webhook retry change cannot degrade the web service
Repository lint: pnpm run lint (declared in package.json) fails over the whole repository (the declared target takes no file arguments), so the failure is not attributed to this change and did not block it; run it before merging.
Fix walkthrough
Watch the fix explained (1:33). Recorded for commit 3c7e57a1c8476962fa605d07d77a7580863d885d.
Full analysis
flowchart TD
A["Manual retry"] --> B["retryCall: call set to PENDING"]
B --> C["enqueueCall(callId)"]
C --> D["queue.remove(callId) clears the retained job"]
D --> E["queue.add(jobId = callId)"]
E --> F["job runs and the event is re-sent"]
G["failed job retained for 30 days"] -. "before this fix add() was a no-op" .-> E
Detection evidence
Affected: int_13ee3c91c001c2c8xfdj06a9
Out of scope
- The per-webhook lock still consumes a delivery attempt on contention. That is why calls fail and reach the retry path, and it is unchanged here.
Production impact
The changed path is the web service's webhook producer (apps/web/src/server/service/webhook-service.ts). The only new runtime effect is a Queue.remove(jobId) before the existing Queue.add; BullMQ's remove returns 0 without throwing for a missing or locked job, so no error path is added. I read the production unsend Railway service (tier 2, serves app.usesend.com) and could not read live metrics because the Railway account returns 403 Not Authorized, so the read is empty.
Affected resources
unsend
Observability
[WebhookQueueService]: Worker errorlogs a failed BullMQ job; the changedenqueueCallis upstream of it.Webhook call <id> completed successfullyandwebhookCall.statustransitions (PENDING, IN_PROGRESS, DELIVERED, FAILED) show whether a retry was delivered.[WebhookQueueService]: Call not foundand the discarded-call log cover the non-active paths.
Polylane assessed this pull request at 3c7e57a.
Causal chain
- Signal (alert): Retrying a failed webhook call never re-dispatches it
- Surfacing site: usesend/useSend web app at apps/web/src/server/service/webhook-service.ts#retryCall
- Mechanism: retryCall resets the call to PENDING and calls enqueueCall, which calls queue.add with jobId=callId. BullMQ still retains the FAILED job under that id for 30 days (removeOnFail age), so add() returns the existing job and stores no new job, and no delivery runs.
- Producer: WebhookQueueService.enqueueCall, at usesend/useSend:apps/web/src/server/service/webhook-service.ts#enqueueCall
- Trigger: Manual retry of a webhook call whose job reached the final attempt and is retained as FAILED.
- What happens to the failed unit today: retryCall exists to re-dispatch a failed delivery; BullMQ add() is a no-op for an existing job id.
- Last observed: 2026-11-16T00:04:16.000Z
- Blast radius: unknown other resource(s), unknown other tenant(s); data at risk: A manual retry never delivers the webhook event; the subscriber misses that event and the call row stays PENDING.
- Producer evidence:
- apps/web/src/server/service/webhook-service.ts#retryCall resets the call and calls WebhookQueueService.enqueueCall(call.id, teamId).
- apps/web/src/server/service/webhook-service.ts#enqueueCall adds the job with { jobId: callId }.
- apps/web/src/server/queue/queue-constants.ts DEFAULT_QUEUE_OPTIONS removeOnFail.age = 30243600, spread into the webhook dispatch queue defaultJobOptions with attempts: WEBHOOK_MAX_ATTEMPTS.
- bullmq 5.79.3 dist/cjs/scripts/addStandardJob-9.js: for a custom jobId that EXISTS, the script returns handleDuplicatedJob, which emits a 'duplicated' event and returns the existing id without storing a new job.
2 files changed (+82/-1)
apps/web/src/server/service/webhook-service.ts: modified, +5/-0apps/web/src/server/service/webhook-service.unit.test.ts: modified, +77/-1
Lint output
ys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
@usesend/ui:lint: $ eslint . --max-warnings 0
@usesend/lib:lint: $ eslint . --max-warnings 0
usesend-js:lint: cache miss, executing 62295f12be2ad8a7
usesend-js:lint: [WARN] The "pnpm" field in package.json is no longer read by pnpm. The following keys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
usesend-js:lint: $ eslint . --max-warnings 0
usesend-js:lint: sh: 1: eslint: not found
usesend-js:lint: [ELIFECYCLE] Command failed.
@usesend/ui:lint: [ELIFECYCLE] Command failed.
Tasks: 1 successful, 3 total
Cached: 0 cached, 3 total
Time: 3.566s
Failed: usesend-js#lint
[ELIFECYCLE] Command failed with exit code 1.
[WARN] The "pnpm" field in package.json is no longer read by pnpm. The following keys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
$ turbo lint
Attention:
Turborepo now collects completely anonymous telemetry regarding usage.
This information is used to shape the Turborepo roadmap and prioritize features.
You can learn more, including how to opt-out if you'd not like to participate in this anonymous program, by visiting the following URL:
https://turborepo.dev/docs/telemetry
usesend-js#lint: ERROR command (/workspace/usesend/useSend/packages/sdk) /root/.local/share/pnpm/.tools/pnpm/11.11.0/bin/pnpm run lint exited (1)
ERROR run failed: command exited (1)
Generated by Polylane. You can ask follow-ups by mentioning @polylane in a comment. If you close this pull request without merging, add a comment that says why. Polylane reads it to make its next fixes better.
Summary by cubic
Retrying a failed webhook delivery returned success but sent nothing. BullMQ keeps failed jobs for 30 days under the call id, and add() ignores a job id that already exists, so the retry never queued a new delivery. The retry now removes the retained job before enqueuing a fresh one, and completed deliveries are unaffected because those jobs are removed immediately.
- Adds
Queue.remove(callId)before the existingQueue.addinenqueueCall, with tests covering the retry and non-retry paths.
