Count each logical batch job's success once

Merged
BI
10h ago
+3214-625

The bug

Consider a batch with two jobs, A and B. A succeeds and the worker records it. The worker stops before it can delete the queue message, so A is delivered again. Recording A a second time takes the pending count from one to zero. The batch runs its completion callback while B is still on the queue. A redelivery that fails can also cancel a batch for work that already succeeded.

There are two ways to see this in practice. The two-job reproduction shows the early callback and traces the same decrement in Laravel's SQL repository. Laravel framework PR #48000 reproduces a negative pending count when concurrent batch retries execute the same jobs. Laravel added command isolation for that case. An earlier upstream report also describes repeated execution driving the count below zero.

The fix

Each logical job can consume one pending slot. Its first success updates the batch. Further successes and failures for that job leave the counters and callbacks alone. A job that failed can still succeed on retry. Repeated failures before that success still contribute to the historical failure count.

Chains need one more distinction. Replaying A can enqueue B twice with two different queue UUIDs. Each enrolled successor now carries a stable position across both dispatches. The queue UUID continues to identify an individual delivery, and the batch retains the UUID of the most recently reported failed delivery so queue:retry-batch and Horizon’s “Retry Failed Jobs” action can retry that delivery.

The accounting transaction finishes the batch and preserves an earlier cancellation time. If work is added before BatchFinished, then, or finally runs, that notification waits for the added work. Calling add([]) leaves a finished batch closed. Callbacks triggered by a success run in the worker that records it. If that worker stops after committing, a redelivery does not replay its callbacks.

Why change the schema?

Adding a success table would fix duplicate counting while leaving an expensive operation in place. With the outstanding failure IDs stored as JSON on the batch row, a successful delivery still has to process a list that grows with every distinct outstanding failure, while other workers wait for that row.

The batch row now holds counters. A separate job_batch_results row records each job that has reported an outcome. Its nullable retry ID represents either an outstanding failure or a completed job. Updating that row and the parent counters together makes the cost of recording a success independent of the number of failures. PostgreSQL handles an ordinary success in one accounting statement. Pruning a batch removes its results, and a batch reopened during pruning keeps its history.

Applications using custom batching configuration need queue.batching.results_table and the Hypervel batch migrations. Read outstanding retry IDs through $batch->failedJobIds. The list loads on first access and has no guaranteed order. Custom batch repositories need the new failure lookup methods and optional retry ID. The dedicated repository methods replace the old parent-only updateAtomicValues() hook. Batch subclasses that override invokeCallbacks() must accept its new optional $expectedTotalJobs argument.

Performance

The table compares the previous Hypervel repository with Laravel-style counting, a duplicate-safe prototype retaining the JSON failure list, and the per-job results design. Figures are aggregate batch bookkeeping operations per second across 50 workers recording 1,000 successes. Each cell shows the median and range across five runs. Shared runs target one batch. In independent runs each worker owns one 20-job batch.

EngineWorkloadPrevious repositorySuccess table + JSON failuresPer-job results
MySQL 9.5Shared, zero failures94 (87–117)97 (75–113)90 (84–119)
MySQL 9.5Shared, 5,000 failures48 (22–58)46 (24–47)91 (82–112)
MySQL 9.5Independent, zero failures1,103 (838–1,216)1,001 (739–1,155)958 (818–1,182)
MariaDB 10.11Shared, zero failures1,023 (1,018–1,041)940 (886–958)959 (934–969)
MariaDB 10.11Shared, 5,000 failures276 (177–278)261 (192–262)938 (145–983)
MariaDB 10.11Independent, zero failures1,824 (344–1,909)1,464 (216–1,665)1,510 (255–1,570)
PostgreSQL 17Shared, zero failures258 (241–267)294 (281–306)292 (275–299)
PostgreSQL 17Shared, 5,000 failures122 (109–129)209 (204–212)292 (276–298)
PostgreSQL 17Independent, zero failures1,552 (1,062–1,622)1,549 (1,497–1,648)1,718 (1,574–1,807)

With 5,000 outstanding failures, per-job results roughly doubled bookkeeping throughput on MySQL, more than tripled it on MariaDB, and improved it by 40% on PostgreSQL compared with the JSON prototype. MySQL's full-row binary log fell from 420,720 to 702 bytes per success. The additional durable write has a cost when jobs succeed on their first attempt: on independent batches, median throughput was 13% lower on MySQL and 17% lower on MariaDB than the previous repository, and 11% higher on PostgreSQL. The measurements cover batch bookkeeping only. The harness, raw samples and methodology are in tests/Benchmarks/JobBatches/.

The queue integration tests replay a chain head, run both copies of its successors, and retry a failed successor after its earlier failed payload has been removed. The database tests also hold a job result open until a competing success times out and retries.

A Redis batch driver

A later PR will add a Redis batch repository. This change gives it the same rules as the SQL implementation: a stable counting ID, a separate retry lookup ID, and atomic updates for each accepted job outcome. Redis can implement those rules with its own storage primitives while keeping the same application-facing batch API.

[!NOTE]

Count each logical batch job's success once via a job_batch_results table

  • Job batches now record per-job outcomes (success and failure) in a new job_batch_results table keyed by batch ID and a hashed job ID, so each logical job is counted once even across chain redeliveries and retries
  • job_batches replaces the failed_job_ids JSON column with an outstanding_failed_jobs counter; Batch.failedJobIds now loads the latest outstanding retry IDs lazily from the results table
  • Chain successors get a stable batchPositionId identity across deliveries; failure accounting stores the delivery UUID as a retry ID so only the latest failed copy is retried (DatabaseBatchRepository.php, Batch.php, CallQueuedHandler.php)
  • Batch deletion and pruning now remove result rows, and BatchesTableCommand generates both the batches and batch-results migrations; queue.batching.results_table is configurable
  • Behavioral Change: failed_job_ids is removed from the schema and BatchRepository gains required methods (getFailedJobIds, hasFailedJob) plus a retry-ID parameter on incrementFailedJobs; custom batch repository implementations must be updated. PostgreSQL uses a dedicated success-accounting statement in DatabaseBatchRepository.php

Macroscope summarized b444364.