fix(web): keep a campaign sending after one of its batches fails

Merged
KM
4d ago
+673-32

What was broken

One failed batch stopped a campaign for good, without a trace.

Each campaign's batches share a fixed BullMQ job id (campaign-batch-<id>), and failed jobs are kept for 30 days. BullMQ ignores add() for an existing id, so after one failure the scheduler (every 1.5s) could never queue the next batch. The campaign stayed RUNNING, and its remaining recipients never got the email. The batch worker had no failed handler, so nothing was logged. Reproduced against real Redis with BullMQ 5.79.3.

The fix

  • Retry after a minute. queueBatch uses requeueJob (from #519). A failed batch is retried after 60 seconds, not on every tick, and logged.
  • Never surprise-restart an old campaign. If a batch failed more than 24 hours ago, the campaign is paused in the database rather than restarted. A pause is durable, where skipping would not be: BullMQ deletes the failed job after 30 days. The pause applies only if updatedAt hasn't changed since it was read, so it can't undo a resume made at the same moment.
  • Explicit actions continue it. Resume, schedule and send clear a kept failed batch. Schedule and send save the new schedule first, so the scheduler can't start an early batch in between.
  • Failures are visible. A failed batch is logged as an error.
  • Cheaper checks. requeueJob now uses isFailed()/isCompleted(), which are single ZSCORE lookups, instead of getState(), which scans the wait and active lists. Webhook Retry benefits too, and its behaviour is otherwise unchanged.

No double sends

A retried batch re-reads the saved cursor and skips contacts that already have a CampaignEmail row. CampaignEmail has a composite primary key, so a second link can't be created. A new test runs the real batch worker: it fails after sending, while saving progress, then runs again. Each contact is emailed exactly once. The test fails if the already-sent check is removed.

Tests

  • requeue-job.integration.test.ts (real Redis in CI): adds cases for "failed too recently" and "failed too long ago".
  • campaign-service.unit.test.ts: retry after a minute; wait when the failure is fresh; pause when it's old, guarded by updatedAt; no pause without a fresh read; resume, schedule and send ordering; the replay test above.
  • Locally: unit (365), tRPC (23) and API (50) tests pass, the integration file passes, and the production build passes.
  • Reviewed by Astra over four rounds. Every finding is addressed, and the final verdict is "merge, no blockers".

No migrations, no schema or config changes.

Supersedes #517 (Polylane), which removed a finished job before every add(). That one would retry a persistently failing batch every 1.5s, could restart long-stuck campaigns by surprise, and used getState() on every tick.

Ask Polylane for an impact assessment of this pull request:

Assess with Polylane


Summary by cubic

Fixes campaigns silently stalling after one of their batches fails, leaving remaining recipients unsent. A failed batch is now retried by the scheduler after a minute, and a campaign whose batch failed more than a day ago is paused instead of being restarted automatically.

Behavior

  • Resume, schedule, and send clear a kept failed batch so the campaign continues.
  • The pause only applies when the campaign's updatedAt is unchanged and the batch is still failed at pause time, so a concurrent resume or reschedule wins; a failure without a finish time counts as fresh.
  • A retried batch skips contacts that already have a CampaignEmail row, so nobody is emailed twice.

Notes

  • Failed batches are now logged as errors; job-state checks use isFailed()/isCompleted() (single ZSCORE lookups) instead of getState(), which also speeds up webhook retry.
  • No migrations, schema, or config changes.

Summary by CodeRabbit

  • Bug Fixes
    • Failed campaign batch jobs are retried after a one-minute delay; failures less than a minute old are left to settle, and failures older than 24 hours are not automatically retried.
    • Campaigns with failures older than 24 hours are paused if they remain scheduled or running and haven’t changed in the meantime.
    • Sending, scheduling, or resuming a campaign clears a retained failed batch job, helping the campaign proceed.
    • Retried campaign work avoids queuing duplicate emails for contacts already processed.