perf(enrichment): poll long-stuck custody deferrals hourly

Merged
GI
2d ago
+136-2

Summary

  • perf(enrichment): poll long-stuck custody deferrals hourly

The enrichment outbox (kind=sync_reprocess on /v2/sync-jobs/run) re-arms a callback custody deferral every
30ā€“120s with no bound. Four reasons can outlive the obligation: redirect_unresolved, conversation_missing,
open_live_target and pending_live_bridge. Once one of those reasons has deferred a row continuously for 6h,
the row polls hourly (custody_defer_until). The outbox records deferred_since for the current reason. A
different reason or a completed callback starts a new episode, and an episode never predates the obligation's
first_pending_at, so a reopened dormant row doesn't inherit an old one. Fresh deferrals and every other reason
keep their cadence. Custody, sweep repair and the eventual resume or redirect are unchanged. No row is retired or
discarded.

Scope: this covers the sync-handler outbox (outbox.run ā†’ defer), which is the serving route
(ENRICHMENT_ROUTE is unset on backend-sync). The C3 consumer (/v2/enrichment/run) is unchanged: its store
records the consumer's own not_due reason, so it needs its own episode field before the route flips. Existing
stuck rows have no deferred_since, so they back off 6h after deploy.

Why (prod measurements 2026-11-01, ro-prod, aggregates only)

  • About 80% of backend-sync /v2/sync-jobs/run 200s carry the 21-byte {"status":"deferred"} body: ~53k
    deliveries/h at a mean of 2.39s, roughly a third of the service's request wall. Measured from the request-log
    census by status and responseSize, 16:10ā€“16:20Z.
  • Open outbox rows by deferred_reason (Firestore COUNT): redirect_unresolved 1,625, pending_live_bridge
    123, open_live_target 83, conversation_missing 12.
  • Ages from a projection histogram of first_pending_at:
    • redirect_unresolved: 98% are older than 6h and 67% are 7ā€“30 days old; 95% are due within 150s.
    • pending_live_bridge: 69% are older than 6h.
  • At the old cadence that is ~49k + ~15k + ~2.5k polls/h. Each poll costs a cohort warm, two outbox CAS writes,
    resolver reads and a new Cloud Task.

Delivery reduction (hermetic)

test_stuck_redirect_deliveries_per_day_drop_from_719_to_198 drives a stuck redirect through the real
outbox.run for 24h of fake time. It makes 719 deliveries before the change and 198 after. The first 181
deliveries (the 6h grace) are identical. After the grace, the largest gap between deliveries is one poll interval
(3600s).

Worst-case added delay: one hour, and only for a row that one reason has deferred continuously for more than 6h.
That covers a redirect that resolves late, or a live target or bridge that settles after 6h.

Review round 1 (Luna, FIX-FIRST)

  • P2 "an old obligation delays a new bridge by up to an hour": fixed. The grace is now per-reason
    deferred_since, not first_pending_at. See test_new_deferral_reason_on_an_old_obligation_restarts_the_grace
    and test_completed_callback_ends_the_deferral_episode.
  • P2 "the sync_hold loop is unchanged": not a live loop. That was an early hypothesis in the context file, and
    measurement then disproved it. Open outbox rows with effects_only=true: 0. Open rows with
    deferred_reason=sync_hold: 0 (Firestore COUNT, 16:45Z). The measured loops are the four reasons covered here.
    Changing sync_hold would add risk with no measured saving.

Lane / contract

Lane syncdiet (two-lane rollout, margin50). Edits fswrites' outbox area: coordinated on the bus (msgs 25ebd3,
1d75c9). The change touches only the defer horizon, not the CAS mechanics.

Product invariants affected

none

Verification

  • backend/test.sh with BACKEND_UNIT_TEST_FILE_LIST: test_sync_enrichment_custody, test_enrichment_route,
    test_outbox_firestore, test_sync_pending_live_bridge, test_sync_reprocess_coalesce,
    test_enrichment_callback_parity and test_sync_processing_hold all pass (unit layer).
  • backend/testing/sync_cloud_tasks_stack/run.sh --only sync passes: the hermetic emulator gauntlet.
  • Not measured: live savings. These need a deploy to backend-sync. Expect the size-121 share of sync-jobs/run to
    fall from ~80% toward ~5%.

Rollback

Revert this commit. Or raise STUCK_DEFER_GRACE_SECONDS in a hotfix. No flag; no data or config changes.

Failure-Class: none

šŸ¤– Generated with Claude Code