perf(enrichment): back off long-stuck custody deferrals on the C3 consumer route

Merged
GI
1d ago
+141-58

Summary

#24264 made long-stuck custody deferrals (redirect_unresolved, conversation_missing, open_live_target, pending_live_bridge) poll hourly after a 6h grace. It only did this in the sync handler's run(). Prod serves enrichment through the C3 consumer (ENRICHMENT_ROUTE=enrichment_queue). There, FirestoreEnrichmentOutbox.claim_key + defer still re-arm the same rows every 30-120s.

This PR brings the same episode rule to the consumer route:

  • Result.defer_cause carries the domain cause. EnrichmentConsumer sets it from the callback's defer_reason, run_push passes it to outbox.defer instead of not_due, and metrics keep ConsumerReason.NOT_DUE.
  • FirestoreEnrichmentOutbox.defer records deferred_since with the shared custody_deferred_since / custody_defer_until helpers, so due_at, task_due_at and wake_check_after move to now + 1h once a cause has held the row for 6h.
  • A completed callback (ack) drops deferred_since, so a new deferral starts a new episode, as in the sync handler.

Custody, sweep repair and the eventual resume are unchanged. The only difference is that a long-stuck row is checked hourly instead of every minute or two. The PG and provider-STT outboxes store the cause string where they used to store not_due; both columns are plain text.

Measurement (prod, ro-prod, aggregates only, 2026-11-02 ~10:40Z)

  • Write probe, last 1h: outbox_firestore:claim_key 81.1k, outbox_firestore:defer 75.6k writes/h (about 157k/h together).
  • sync_reprocess_outbox rows with deferred_reason == not_due: 7,554. Rows waking within 2 min (the hot loop): 1,800 are ā‰„6h old and about 170 are newer. 4.1k old rows already wake more than 1h out.
  • Every consumer custody cause (utils/enrichment/callback.py) is in STUCK_DEFER_REASONS.
  • Expected effect: about 91% of the hot loop goes from roughly 40 cycles/h to 1, so claim_key+defer should fall from ~157k/h to ~20k/h. That is about 3.3M writes/day ā‰ˆ $4.6/day at the billed $1.385/M, plus about 3.3M reads/day ā‰ˆ $1.5/day, along with Cloud Tasks deliveries and consumer CPU.
  • Timing: existing rows have no consumer deferred_since, so each episode starts at the row's first deferral after deploy. The drop appears about 6h after the deploy.
  • Verify with omi_firestore_billed_writes_by_caller_total{caller=~"database.outbox_firestore:(claim_key|defer)"} from deploy+6h onward.

Rollout

Ships on by default. It reuses #24264's constants, so there is no new flag. Deploy: backend-integration (the prod /v2/enrichment/run target, API_WORKERS role). The lead batches it into a train.

Tests

backend/test.sh with the targeted list (outbox_firestore, enrichment_route, workers_consumer_contract, sync_enrichment_custody, sync_reprocess_coalesce, enrichment_callback_parity, sync_processing_hold, sync_pending_live_bridge, live_demand_consumers, feedback_media_consumers): all pass. make preflight: passed.

šŸ¤– Generated with Claude Code