[13.x] Fix race condition with jobs using both `onOneServer()` and `withoutOverlapping()`

Merged
TA
19d ago
+48-3

Fixes issue https://github.com/laravel/framework/issues/50330

When the scheduler runs on several servers, a job using both onOneServer() and withoutOverlapping() is occasionally skipped on every server.

This was happening on a project about 1-3% of the time, pretty substantial!

Background: CacheEventMutex::exists() checks whether a task is running by acquiring the event lock and releasing it.

Failure mode: while checking for the lock's existence, the lock is acquired momentarily, and other servers that try to run the job think that the the job is "overlapping". See the issue for a more detailed explanation and example.

Change: This changes exists() to use Lock::isLocked() (added in April this year in PR #59791), which gets the status of the lock without acquiring it.

Previous attempts by others:

  • PR #54430 did not handle locks that were not stored in the same cache store (typical for database cache using cache_locks table instead of cache; or Redis using lock_connection)

Thanks @nickma42 for the investigation, and @jackbayliss for the great isLocked() addition!

Disclaimer: arguably ShouldBeUnique was the better solution for the root issue here, but this seemed like an easy fix, and it might easily bite others too. This issue might be under-reported as the failure is silent, the typical way you would find out is if you were monitoring the job runs.