Monitoring: alerts cover the full 6 hours, a paused worker is visible, every alert pings the owner

Merged
BI
23d ago
+1653-180

Summary

Before: answer-quality alerts judged only the last hour of each 6-hour digest and needed 20 answers in that hour, so at current traffic they never fired. The live digest on 20 Oct saw 0 answers. A stopped or paused worker was invisible, and pending jobs of any age counted as healthy. Most alerts did not @mention the owner on repeats. Canary results disappeared after 30 days. A cancelled or timed-out canary raised no issue and still counted as recently run.

After:

  • Answer alerts judge the trailing 6 hours, using a new 360-minute health snapshot. An older build without it falls back to the hourly snapshot. The minimum answer count is now 10 rather than 20, and the reason is in a comment. Other thresholds are unchanged.
  • Pending jobs left unclaimed for more than 30 minutes raise an alert. The worker shows as "paused" in the autopilot summary and the digest, and a pause alone does not page the owner.
  • Eight alert workflows now @mention the owner in every new alert and repeat comment, and the contradictory doc text is corrected.
  • Every canary run, cancelled ones included, posts one row of numbers and case IDs to a rolling "Eval canary trend" issue and to the job summary. Nothing is committed from CI.
  • Cancelled or timed-out canaries open the failure issue as "not completed". The digest's liveness check counts only runs that reached a pass or fail.

Fixes audit findings EVAL-02, EVAL-03, EVAL-04, EVAL-06 and EVAL-08 from the 20 Oct 2026 RAG and ingestion audit. EVAL-05 (numeric golden expectations) is deliberately not done: no offline evidence proves the live passage text, and clinical values need the owner's sign-off.

RAG impact: no retrieval behaviour change - monitoring, alerting and canary reporting only.

Required check change approved: owner's yes by decision card on 21 Oct 2026, recorded in docs/decisions/2026-10-21-ci-failure-alerts-mention-owner.md (ci.yml alert wording only, no required-check change)

Clinical Governance Preflight

  • Source verification: no change to sources, retrieval or answers.
  • Patient-data handling: the alerts and trend rows carry counts, rates and synthetic golden case IDs only. No question text is included.
  • Service-role confinement: the existing health deep-probe gains a 360-minute window over the same aggregate reads. No new access is added.

Verification

  • Before and after: with the old workflows swapped back in, 6 tests fail across eval-canary-trend, eval-canary-workflow and ops-digest. With the fix, all 69 pass.
  • Unit suite: 1 failed | 33053 passed. The one failure was the CI cache-safety registration of the new workflow tests, fixed in 6d7440cdc. That test then passed 82 of 82. The whole suite was not rerun after that commit.
  • npm run lint passed and npm run typecheck passed.
  • check:diff-integrity: PASS ā€” 150 -> 182 test case(s).
  • After merging main (b074e80d3, live-drift.yml conflict resolved to one owner mention): the 9 monitoring and alert test files, Tests 234 passed (234).
  • tests/eval-canary-workflow.test.ts was rewritten for the new behaviour. The commit says so.

Risk and rollout

  • Risk: more notifications. Every red CI run on main will now also @mention the owner.
  • Operator step: set the GitHub repository variable INGESTION_WORKER_PAUSED to match the worker's real state. Until it is set, the digest shows "state not recorded".
  • Unverified: whether a job-level timeout actually runs the cancelled() steps. The liveness change covers that case anyway.
  • Rollback: revert the PR.
  • Provider or production effects: None at merge. The workflows run on their existing schedules.

Generated by Claude Code