fix(batch): bound held dwell and re-drive retained dead letters

Merged
GI
4h ago
+1458-17

GPU batch holds had no bounded terminal disposition or automatic re-attempt. This adds a seven-day held_deadline_at, retained dead(held_expired) rows, a private-OIDC CAS requeue, a nightly UTC re-drive on the existing serving maintenance leader, and an agent-tier volume alert. No CronJob is added.

Initial quarantine and failed-held recovery writers stamp a full window. Legacy rows receive seven days at their first eligible sweep. Expiry/requeue recheck content ownership, privacy generation, dormant held state, no result or result receipt, no worker token and an expired finite lease. Active paid/operator intents block the transition. Durable document-key cursors bound each pass to 32 rows per five-minute tick; crashes replay idempotently and malformed rows cannot pin the page. Nightly re-drive rechecks its previous-day cutoff inside the transaction before normal GPU admission.

BATCH_HELD_DEAD_LETTER_ENABLED / batch_held_dead_letter_enabled() defaults ON. It is registered in kill switches, flag budget, feature registry, deployment classification, common runtime declarations and worker overlays. Rollback: false on serving/workers stops new stamps and automatic maintenance; the private requeue endpoint remains usable. Expiry atomically sets merge_pending=true, so after reverting the new code an operator can still use the unchanged POST /v2/batch-pipeline/redrive-dead route. No pre-revert drain or data rewrite is required for rows written by this implementation. The rolled-back service requires an operator re-drive rather than the new nightly helper. See backend/docs/runbooks/batch-held-dead-letter.md.

Custody: nothing is deleted. Every new dead letter has reason held_expired, retains its deadline, encrypted audio references and attempt/intent history, and has a sanctioned requeue path. No TTL, conversation-status writer or custody/privacy fence is changed. Status and partial-accounting readers retain the held answer, so age does not acknowledge phone custody or hide healthy siblings. Projection expiry (4b), BYOK/finalization tools (4c), and privacy_blocked remain outside this change. Provider-STT holds remain fenced from the GPU queue; the state-table deadline gap remains explicitly strict-xfailed.

The volume alert is agent tier, with omi-tier-silent routing so it never pages humans. It checks fresh retained stock >100 or >10 expirations in 30 minutes for five minutes. Each stock sample must be younger than 15 minutes before aggregation; unrelated legacy dead reasons do not count. Notification timing follows the warning contract. Split/combined rules, metric inventories, the sync GMP scrape allowlist and all four dev/prod main/sync exporter prefix lists include the three emitted metric families. Thresholds are initial operator defaults, not production-calibrated. Backend and metrics-egress/alert rollout must use the checked-in declarations; this PR performs none of those rollouts.

Validation (offline/synthetic; suites overlap):

  • Final held-recovery, alert, notification, inventory and transport contracts: 319 passed.
  • Real loopback Firestore integration: 3 passed, including expiry → flag off → unchanged legacy dead re-drive → normal completion/merge, contention and duplicate delivery.
  • Main-sync serving/flag/held/worker contracts: 135 passed.
  • State-table/writer/policy validation retained 75 strict expected gaps; no unrelated xfails were removed. The shared emulator regressions are also registered in the lifecycle harness.
  • make preflight and scripts/pr-preflight: 52 checks passed. Both published fix and merge heads passed the normal pre-push hook, including backend tests, type checks, OpenAPI and Firestore guard; no bypass, cap override, force or timeout retry.
  • Required Grok review completed with NO BLOCKERS (receipts exec-heavy-system-squeeze-major-install-point, exec-breeze-portion-boy-castle-gauge-marine, exec-blur-source-live-meadow-inject-damp). The last Luna P1 rollback finding is fixed inline with the atomic legacy marker and explicit rollback regressions; no further review round was requested.
  • Exact-head GitHub checks for 30b59b4c95631168a4a5fac9d7807cadd747327c.

Main is merged through d6112658e5fa, retaining #24735's cadence controls and #24733's 16-worker seal-repair rollback. The lane base already includes #24712/#24716. No .github edits; unrelated Flutter work is preserved.

Not measured: production stock/dwell, vendor spend, live nightly execution, live exporter/alert delivery, deployed revision or user-visible recovery. No GCP tier, production access, merge to main or deployment was used.

Product invariant: INV-MEM-4; canonical memory intake/promotion authority is unchanged.
Failure-Class: FC-batch-fault-domain-head-of-line-poison