fix: preserve timeliness metrics when Cloud Run integration leads
Cloud Run backend-integration shares the timeliness lease with GKE. Its collector and bridge must retain all sampler families, and alerts must reject a superseded leader's replayed gauges.
Head: 264a2ebd46062b032a406a9571f20e465994f272
Failure-Class: FC-metric-dropped-by-scrape-allowlist
Keep all producer-registered timeliness families in the sync GMP profile and dev/prod bridge. Each alert value joins its own observation with ignoring(leg), retaining every source identity label, then selects sources tied for the newest numeric observation before aggregation. Election and sampling behavior are unchanged.
Alert expressions
| Source | Rule UID / refs |
|---|---|
alerts/batch-timeliness.json | omi-batch-unit-unpublished-2h: A/B/D; omi-batch-timeliness-sample-stale: A/D; omi-batch-timeliness-scan-blind: A |
alerts/batch-input-retention.json | omi-batch-staged-input-age: A |
alerts/pipeline-drain.json | omi-pipeline-projection-drain-stall: C; omi-pipeline-held-row-age: A |
alerts/critical-failures.json | omi-batch-held-backlog-growing: A, including growth subquery and saturation |
The combined import and migration inventory mirror these expressions. Per-source max_over_time(observed_at[20m]) bridges sparse-point lookback gaps; 1200s freshness covers the 600s maximum cadence + 60s collection grace + about 180s delivery delay, with 360s jitter margin. The stale rule retains its 5m pending period. All threshold annotations and import-disposition expectations agree with the executable 1200s conditions.
The four timeliness prefixes are pinned in both exporter contracts; the prefix-overlap checks remain intact. Producer names drive export and consumer coverage. Pure-Python CI tests check every consuming expression's own-source ignoring(leg) pairing and newest-observation selector, then mutate every consumer to prove missing selection, wrong source identity and unpaired values are rejected. Existing test-selection coverage and full-suite fallback are preserved. No .github changes.
Focused validation
Documented backend/test.sh runner through omi-heavy, with promtool unset as in CI: 196 passed, 1 skipped across exporter telemetry (56), production import disposition (7), structural/export guard (52 plus the intentional local-tool skip), monitoring alert contracts (75), and notes prompt-version checks (6). The pure-Python structural and mutation tests run in CI. The promtool fixture is explicitly local-only, opts in through PROMTOOL_BIN, and its skip reason states that CI does not run it.
Explicit local check, promtool 2.55.1 installed outside the repository:
PROMTOOL_BIN=/Volumes/Ephemeral/scratch/tokenlease-promtool/promtool \
PYTEST_ADDOPTS='-s -k test_prometheus_replayed_handoffs_and_sustained_leadership' \
BACKEND_UNIT_TEST_FILE_LIST=<one-file-list> bash backend/test.sh
Passing output from the committed fixture on this head:
1 passed, 52 deselected in 2.83s
The fixture invoked promtool test rules on its generated YAML on this merge head (exit 0):
SUCCESS
This evaluates both replayed handoff directions, every consuming split expression, sustained 600s observations with 180s Cloud Run delay, and a stopped sampler that still pages.
Main synchronization: merged current origin/main normally, without conflicts. This includes the notes prompt-version fix verified by green main Backend Unit Tests run 43000370042. No rebase or force-push. The previous notes-digest blocker is resolved by upstream; the focused suite and explicit local promtool fixture pass on this merge head. Backend Unit Tests run 43006342683 passed every shard (1/8, 2/8, 3/8, 4/8, 5/8, 6/8, 7/8, 8/8) and backend guardrails on this exact head. The previous shard-3 notes-digest failure is cleared.
Post-deploy qualification requires a sustained Cloud Run-led window, covering multiple sample intervals and pending windows in GMP and the Prometheus alert datasource, followed by both handoff directions. A first visible sample is insufficient. No merge, deployment, GCP tier or production-account access.
Product invariants affected
none
š¤ Generated with Claude Code