fix(batch): collapse projection pages and prioritize productive drains

Merged
GI
15h ago
+306-50

Problem

Heavy owners fill projection discovery pages with non-head rows. Each row repeats
full projection reads and predecessor checks even though only the chain head can
advance. Productive 120-second drain turns then rejoin behind every waiting owner.

Fix

  • Group page job hints by durable binding owner before projection attempts. The
    assignment sweep attempts one page head per owner, counts collapsed rows as
    waiting, and requests that owner's drain once.
  • Offer one follow-up receipt per owner per page, rotating within each owner's
    page slice on later passes so ready receipts cannot pin their unfinished
    siblings, including owners spanning several pages. Rotation state is bounded.
  • Requeue productive deadline/budget turns at the head at most twice, then yield
    to the tail. Zero-advance turns retain FIFO behavior. Owner claim leases,
    predecessor fencing and transactional ordering remain unchanged.
  • Preserve the existing follow-up-page kill switch's legacy read budget; restart
    recovery tests verify receipt readiness over bounded owner-collapsed passes.
  • Export collapsed-row and head-requeue counters through the existing metrics
    module and retain both metric families in the serving GMP profile.

Tests

cd backend && .venv/bin/python -m pytest with -q --tb=short for these files,
through bin/omi-heavy run --: 359 passed (including the slow simulation tests).

  • tests/unit/test_batch_pipeline_owner_drain.py
  • tests/unit/test_batch_pipeline_projection_followup.py
  • tests/unit/test_batch_pipeline_projection_discovery.py
  • tests/unit/test_batch_pipeline_owner_chain_projection.py
  • tests/unit/test_batch_pipeline_metrics.py
  • tests/unit/test_batch_pipeline_sweep_owner_chain.py
  • tests/unit/test_batch_pipeline_followup_desync.py
  • tests/unit/test_batch_pipeline_projection.py
  • tests/unit/test_batch_pipeline_projection_repair.py
  • tests/unit/test_pipeline_sim_projection_drain.py
  • tests/unit/test_batch_pipeline_followup_reads.py
  • tests/unit/test_batch_pipeline_recovery_owner_only.py

The focused follow-up/read-budget/restart/desync run also passed: 121 passed.
BACKEND_UNIT_FILE_BUDGET_SECONDS=900 OMI_PR_BODY_FILE=<this body> make preflight
through omi-heavy: 36 checks passed. PR metadata validation: 4 checks passed.
The longer supported file budget accommodates CPU contention on the shared host.

Risk and rollback

The ID-only discovery API still requires one job hint read per page row; collapsed
non-heads avoid full projection reads and predecessor transactions. A page head
may be beyond the true chain head; the existing predecessor replay resumes from
the durable watermark. Bounded head requeues trade at most two extra turns of
latency for other queued owners against faster productive chain continuation.

No new flag. Revert this commit to roll back; obligations remain durable. The
existing owner-drain switch restores the sweep walk path. No production access,
deployment, real Firestore integration run or production throughput measurement
was performed. The operator repair script is untouched.

Failure-Class: FC-bounded-discovery-pinned-by-blocked-page

Line-Count-Exception: backend/utils/batch_pipeline/projection.py | 1698 -> 1712 | Keep the small pre-attempt page collapse at the sweep boundary; shared grouping lives in projection_discovery.py.