fix(web): keep a campaign sending after one batch job fails

Closed
PO
4d ago
+69-2

A campaign can silently stop delivering when one of its batch jobs fails. The batch is queued under a fixed job id, and BullMQ keeps a failed job under that id for 30 days, so the scheduler can never queue the next batch and the remaining recipients never get the email. After this change the next batch is queued and the campaign runs to completion. Inspected code shows a failed batch job under the fixed id makes every later add() a no-op, and a campaign advances only through that batch worker, so one transient failure stops delivery for good. Clearing a finished job before the enqueue lets the scheduler continue, and it leaves a batch that is still running untouched.

Production impact at 1eebff9: Campaign enqueue restores only a finished job, no production regression

Repository lint: pnpm run lint (declared in package.json) fails over the whole repository (the declared target takes no file arguments), so the failure is not attributed to this change and did not block it; run it before merging.

Fix walkthrough

Watch the fix walkthrough

Watch the fix explained (1:09). Recorded for commit 1eebff9b8c8ecc4e5143db004c32e25f187ee323.

Full analysis
flowchart TD
  A["Scheduler tick ~1.5s"] --> B["queueBatch: add jobId campaign-batch-ID"]
  B --> C{"Job key exists?"}
  C -- "no" --> D["Worker processes one batch, advances lastCursor"]
  D -- "ok" --> E["removeOnComplete frees the id"]
  E --> F["Campaign reaches SENT"]
  D -- "throws" --> G["Job FAILED, retained 30 days"]
  G --> C
  C -- "yes" --> H["handleDuplicatedJob: add() no-op"]
  H --> I["Campaign stuck in RUNNING, contacts unsent"]
  J["Fix: queueBatch clears a completed or failed job before add()"] -.-> B
Detection evidence

Affected: int_13ee3c91c001c2c8xfdj06a9

Out of scope
  • The same retained-job-id pattern in apps/web/src/server/service/webhook-service.ts and email-queue-service.ts; each is a separate producer tracked on its own issue.

Production impact

I checked the deployed Railway service unsend (production, serves app.usesend.com), which this repository deploys to. The change adds a getJob/getState read and clears a job only when it is completed or failed, then enqueues the next batch; an active or waiting batch is left untouched, so an in-flight batch is never disturbed and no extra load is added. Live metrics are frozen at 2026-10-21 because the Railway reads return 403, so the production observability read is empty.

Checked and not affected (2)
  • unsend
  • useSend

Observability

  • The change adds no new signal; it uses the existing getJob/getState/remove calls on the BullMQ campaign queue.
  • Operators see the corrected behavior in the BullMQ campaign queue: no campaign-batch-<id> job is left in failed state while its campaign stays RUNNING, and campaign.lastSentAt keeps advancing.
  • campaign-scheduler-job.ts logs its tick, and campaign-service.ts logs the defensive window skip, using the repository's existing pino logger.

Polylane assessed this pull request at 1eebff9.

Causal chain
  • Signal (alert): Campaign sending stalls permanently after one failed batch job
  • Surfacing site: campaign batch queue (BullMQ) in the useSend web app at usesend/useSend:apps/web/src/server/service/campaign-service.ts#CampaignBatchService.queueBatch
  • Mechanism: queueBatch adds the batch with the fixed jobId campaign-batch-${campaignId}. When a batch job throws, createWorkerHandler rethrows, BullMQ marks it FAILED, and DEFAULT_QUEUE_OPTIONS retains it for 30 days. BullMQ's addStandardJob script returns handleDuplicatedJob when the job key exists in any state, so every later add() stores no job and the scheduler never refills the batch.
  • Producer: CampaignBatchService.queueBatch on the BullMQ campaign queue, at usesend/useSend:apps/web/src/server/service/campaign-service.ts#CampaignBatchService.queueBatch
  • Trigger: Any batch job that throws: a DB/Redis error in db.contact.findMany or db.campaign.update, or another unexpected worker error.
  • What happens to the failed unit today: DEFAULT_QUEUE_OPTIONS: removeOnComplete true, removeOnFail age 30 days, no attempts; the fixed jobId is used to serialize one batch per campaign.
  • Cadence check: Scheduler refills a batch about every 1.5s; the fixed jobId blocks each refill once a failed job is retained.
  • Last observed: 2026-11-16T17:30:06Z
  • Blast radius: unknown other resource(s), unknown other tenant(s); data at risk: Emails to the remaining subscribed contacts of a stalled campaign are delayed indefinitely; the campaign stays in RUNNING.
  • Producer evidence:
    • apps/web/src/server/service/campaign-service.ts: add(campaign-${campaignId}, ..., { jobId: campaign-batch-${campaignId}, ...DEFAULT_QUEUE_OPTIONS })
    • apps/web/src/server/queue/queue-constants.ts DEFAULT_QUEUE_OPTIONS: removeOnComplete true, removeOnFail age 30 days, no attempts
    • node_modules/.pnpm/bullmq@5.79.3/.../addStandardJob-9.lua:90 EXISTS jobIdKey then handleDuplicatedJob; handleDuplicatedJob.lua returns jobId without storing
    • apps/web/src/server/jobs/campaign-scheduler-job.ts is the only refill path; apps/web/src/server/queue/bullmq-context.ts createWorkerHandler does not catch, so a worker throw marks the job FAILED
2 files changed (+69/-2)
  • apps/web/src/server/service/campaign-service.ts: modified, +17/-1
  • apps/web/src/server/service/campaign-service.unit.test.ts: modified, +52/-1
Lint output
on is no longer read by pnpm. The following keys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
@usesend/lib:lint: $ eslint . --max-warnings 0
usesend-js:lint: cache miss, executing 62295f12be2ad8a7
usesend-js:lint: [WARN] The "pnpm" field in package.json is no longer read by pnpm. The following keys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
usesend-js:lint: $ eslint . --max-warnings 0
usesend-js:lint: sh: 1: eslint: not found
usesend-js:lint: [ELIFECYCLE] Command failed.
@usesend/ui:lint: [ELIFECYCLE] Command failed.

 Tasks:    1 successful, 3 total
Cached:    0 cached, 3 total
  Time:    4.395s 
Failed:    usesend-js#lint

[ELIFECYCLE] Command failed with exit code 1.
[WARN] The "pnpm" field in package.json is no longer read by pnpm. The following keys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
$ turbo lint

Attention:
Turborepo now collects completely anonymous telemetry regarding usage.
This information is used to shape the Turborepo roadmap and prioritize features.
You can learn more, including how to opt-out if you'd not like to participate in this anonymous program, by visiting the following URL:
https://turborepo.dev/docs/telemetry

usesend-js#lint:  ERROR  command (/workspace/usesend/useSend/packages/sdk) /root/.local/share/pnpm/.tools/pnpm/11.11.0/bin/pnpm run lint exited (1)
 ERROR  run failed: command  exited (1)

View autofix


Generated by Polylane. You can ask follow-ups by mentioning @polylane in a comment. If you close this pull request without merging, add a comment that says why. Polylane reads it to make its next fixes better.


Summary by cubic

A campaign can silently stop delivering when one of its batch jobs fails. BullMQ keeps the failed job under the fixed campaign-batch-<id> key for 30 days, and every later add() becomes a no-op, so the scheduler can never queue the next batch and remaining recipients never get the email. This change clears a retained finished job before enqueueing the next batch, letting the campaign run to completion. An active or waiting batch is left untouched, so an in-flight batch is never disturbed.

  • Clears only completed or failed jobs, verified via getJob/getState before remove.