fix(batch): make owner drains claim-aware with contention backoff

Merged
GI
9h ago
+353-14

Problem

Fleet instances repeatedly request the same projection owners. Requests arriving during a losing claim-busy turn immediately re-enter the FIFO and repeat the contention.

Fix

  • Read-only GET of the existing Redis owner-claim key at initial launch, off the event loop; held owners defer admission and any peek error admits as before.
  • Jittered 5ā€“15 second backoff only for claim_busy turns re-requested through again. Productive more=True continuations retain immediate FIFO progress and bypass the new admission peek.
  • Cap retry timers and clear them on close; return overflow intent to the existing next-serving-pass handoff.
  • Export skipped-drain and claim-busy-backoff counters through the serving GMP keep-list.

Based on merged revert #24707 (e88fb828); only L2 is included. Prefetch, capacity, page sizes, assignment fencing and the #24697 collapse remain unchanged. The optional head item was skipped: the cited code is a read-before-owner-draining skip, not a clear head-staleness skip.

Tests

  • Requested stage-env/budget/owner-drain guard through BACKEND_UNIT_TEST_FILE_LIST=/tmp/l2f.txt bash test.sh: 97 passed across 3 isolated files.
  • Original focused owner-drain/projection/simulation/sweep/metrics/GMP set: 426 passed.
  • Policy suite: 73 passed; switch semantics cover shadow and serve parser modes.
  • Registry generator/parser check passed; stage-env ratchet baseline is unchanged.
  • The switch is now read through config.batch_pipeline / config.policy.kill_switch, with an owned helper-backed flag-budget entry and generated registry documentation. Only off disables it in every parser mode.
  • Final repository gates and exact-head GitHub CI are being rerun for the follow-up commit.
  • Code and hermetic unit tests only; no production access or GCP tier used.

Risk and rollback

Low-risk advisory scheduling change; durable result rows remain discoverable. PROJECTION_DRAIN_CLAIM_AWARE=off restores previous admission, re-request and FIFO behavior. Default is on; only off disables it.

Failure-Class: FC-immediate-retry-loses-shared-cas

Line-Count-Exception: backend/config/batch_pipeline.py | 818 -> 825 | Add the requested seven-line policy kill-switch accessor to remove the raw stage env read; no stage-env ratchet baseline change.