fix(batch): add proof-gated exits for aged record-pending jobs

Merged
GI
16h ago
+959-36

Problem

Seal repair discovers completed conversations, so aged record_pending jobs with stuck producers, failed uploads, closed records, partial barriers or missing conversations can remain invisible or refused forever.

Fix

Add a default-on jobs discovery leg (RECORD_SEAL_REPAIR_JOBS=off disables it) with an independent encrypted cursor. Its 128-document page shares the existing 10,000-read/1,024-row/60-second discovery bounds and existing row/run deadlines. Use the existing 30-minute partial wait; zero-second upload admission grace would attempt repair too early.

Reuse fresh producer inventory, full retained-tail proof, same-digest apply and transactional privacy checks. Explicit closed-row repair still proves both current and frozen finishes; admission owns job advancement. Retry aged partial barriers after the existing one-hour inventory interval through finish_partial, preserving the barrier and generations. Aged retired coordinators rearm through the same fresh-proof transaction.

After seven days, transactionally cancel only canonical pending-job metadata when the conversation remains absent. Privacy-fenced cancellation is revocation-only, like cancel_fenced_recovery; no speech receipt, audio deletion, TTL or conversation write. Add closed blocker/outcome log counters and documented rollback.

Tests

  • B1/B2 review fixes: four affected suites (pending repair, partial, cluster repair, chart rendering) — 205 passed. Tests cover proof-gated retired rearm, retirement during proof, rollback suppressing jobs queries and partial selection/retry, rollback during proof, and default-on/off chart rendering.
  • cd backend && /Users/dazheng/workspace/omi/bin/omi-heavy run -- .venv/bin/python -m pytest tests/unit/test_record_pending_repair.py tests/unit/test_record_seal_cluster_repair.py tests/unit/test_record_partial.py tests/unit/test_record_seal_repair.py tests/unit/test_record_seal_nojob.py tests/unit/test_record_seal_cluster_repair_deploy.py -q — 281 passed.
  • python -m pytest scripts/test_check_feature_flag_registry.py -q — 46 passed; registered default-on switch and regenerated catalog.
  • Firestore query guard — all seven files passed, including 549 query-shape tests; the new builder uses the existing state/name index.
  • After Pyright input narrowing, reran both affected backend suites — 124 passed.
  • Guarded push: all 41 shared preflight checks and Pyright (zero errors) passed; the bounded selected backend test runner passed.
  • One parallel independent review round; shared discovery-budget finding corrected and covered by an assertion.

Risk and rollback

No custody owner or direct job admission change. Missing-tail/live/unknown-producer/privacy refusals remain protected. Unfinished conversations are discovered but retain not_completed; finalization stays with its current owner. Seven-day cancellation is terminal only for missing-conversation job metadata and does not claim no speech. No production access or live verification; no merge or deploy.

Set recordSealRepair.jobsDiscovery=off to wire RECORD_SEAL_REPAIR_JOBS=off into the CronJob and disable both the added jobs leg and aged-partial retries, restoring prior discovery and partial-retry behavior. Existing top-level repair switch also stops the coordinator.

Failure-Class: FC-bounded-discovery-pinned-by-blocked-page