#3751 Answer quality: trend line, time to first content, metabolic monitoring answer

Requested by Josh · project thread

Summary

Master plan section 4 (Answer quality), one PR. The answer fix commits can be reverted alone.

• Quality trend line (scripts/lib/answer-quality-line.mjs): reads the rows the eval canary already posts to the rolling "Eval canary trend" issue (#3712) and returns one plain line for the daily health summary: answers passing, direction against the run before, search pass count and recall, the last eight runs, and failing case ids. Counts and case ids only.
• Time to first visible content: the route deadline stamps the moment the verified evidence preview is handed to the stream, and rag_queries.metadata gains first_verified_content_ms beside total_latency_ms when a preview was sent (audit DOC-11). The SLO snapshot adds timeToFirstContent p50/p95, and /api/health?deep=1&answer_window=daily adds a 24-hour sloDaily block. scripts/lib/answer-speed-line.mjs turns that into the summary's speed line. Display only, with no new alert.
• Daily summary wiring: the section 5 stub (scripts/lib/daily-health-summary-answer-lines.mjs) is on main now, so it calls both readers, passing the workflow's owner/repo and turning its ISO time into a Date.
• Extractive answer text (#J8SJQ9, #E9F5GW), from diagnostic Eval Canary 39614340500:
  • The metabolic monitoring answer led with the screening policy's "Compliance, monitoring and evaluation" section, then with its glossary row, while the schedule sat in a lower-ranked source. Monitoring-schedule facts now reject a policy's own governance and glossary sentences (policyAdministrationSentencePattern, glossaryEntrySentencePattern), and a single medicine class ("antipsychotics") becomes the subject instead of "metabolic". A question naming two classes gets no class subject.
  • The lithium level timing fallback shipped "The guidance for lithium is that therapeutic Drug Monitoring. Blood taken ...". completeExtractiveSentence now drops a short Title Case heading glued to a complete sentence, rather than wrapping it.
  • The eval case quality-antipsychotic-metabolic-monitoring now requires a monitored measure (weight, BMI, waist, glucose, HbA1c, lipids or blood pressure), because the glossary answer passed its old keyword cue.

Areas touched: Answer engine, Data platform and access, App experience, Source intake and indexing (plus 2 files not yet placed in any area)
RAG impact: behaviour change — canary pair 39614340500 -> 39711681854 (extractive sentence selection for monitoring-schedule answers and the heading wrapper; no ranking, retrieval or comparator change)

After run (Eval Canary 39711681854, main dcf150865)

• Retrieval identical to the before run: 36/36, document and content recall 1.0, zero per-case reciprocal-rank change.
• Answer subset 42/44 (before 43/44). The one new failure is neuroleptic-side-effect-escalation at 21.7 s against a 20 s limit, on the model route this PR does not touch (generation 17.2 s against 14.7 s before).
• Answer targeting 0.50 (before 0.42).
• The answer fixes landed only in part: the metabolic answer gained "weight at every visit" and "waist circumference" lines, but still opens with a glossary row ("Assessment Tool An evidence-based tool is utilised …") and now labels a lithium sentence "For antipsychotics". The lithium timing opening is unchanged. Follow-up work is tracked in the project thread; clinical sign-off waits for it.

Verification

• New tests/rag-monitoring-policy-administration.test.ts. 4 of its first 5 tests fail on the unfixed source.
• Full unit suite: 33,317 passed and 2 failed. Both failures are in tests/test-runner-safety.test.ts: Playwright browser revision drift in the cloud container, unrelated to this diff.
• npm run lint and npm run typecheck are clean. rag.ts stays within its line budget.
• tests/daily-health-summary.test.ts covers the wiring (repository and ISO time handed through).
• Before baseline (Eval Canary 39614340500, main 8c03823aa): retrieval 36/36, answer subset 43/44, targeting 0.42.

Risk and rollout

• Risk: telemetry, reporting and wiring are low risk (read-only). The answer commits change which source sentence leads some monitoring answers. The filters are narrow (policy compliance and glossary wording, and Title Case headings of up to 6 words).
• Rollback: revert the answer commits alone if needed.
• Provider or production effects: three owner-approved Eval Canary runs, capped at US$3 each, about US$2.30 in total.
• Clinical sign-off: Josh signs off on the metabolic and lithium answer text once the follow-up lands.

---

Generated by Claude Code

🤖 Generated with Claude Code

https://claude.ai/code/session_01CEnwd6W6Y8NjEj7wNgs1n7