Answer safety hardening: source tier rule, superseded sources out of evidence, national guidance line, tests and ledger

Merged
BI
23d ago
+930-35

Requested by Josh · project thread

Summary

  • tests/australian-source-priority.test.ts: pins the owner tier rule from #BFCTST. At equal relevance the order is WA validated, then Australian national, then other states, then supplementary. Relevance always wins across bands. Supplementary padding is dropped once two Australian documents suffice.
  • tests/registered-public-answer-options.test.ts (new): synthetic gating tests for src/lib/registered-public-answer-options.ts (audit DOC-18).
  • Superseded sources out of evidence (owner rule 2026-09-25, approved for build 2026-10-21):
    • src/lib/rag/superseded-evidence.ts removes document_status: "outdated" rows once in src/lib/rag/rag.ts, before every answer route.
    • review_due and unknown are unchanged.
    • A read-only live count on 2026-10-21 found 0 indexed documents marked outdated (2185 current, 588 review_due, 80 unknown), so no current answer should change.
    • Two fallback tests used an outdated row only to trigger the governance-gap marker. They now use poor extraction, which raises the same marker.
  • National guidance line (owner decision 2026-10-21; mockup approved on the owner's PC, Live site visual checks item 8):
    • When a WA guideline and a national guideline differ on a point, the answer still follows WA. One plain grey line under the prose reads "National guidance differs on this point. See
    • src/lib/rag/guidance-difference-note.ts attaches it in src/lib/answer-response.ts. It is passed through clientAnswerFieldsSchema and rendered by NaturalLanguageAnswer.
    • The line only comes from an owner-signed entry in src/data/reviewed-guidance-differences.json. That file ships empty, so no live answer changes until the owner signs off a difference.
    • Partial support, refusals, ungrounded answers and expired entries never show the line.
  • tests/fixtures/kemh-lithium-breastfeeding-live-excerpt.json: KEMH passage captured read-only for #ZZ4RAP case breastfeeding-lithium. It is not wired yet because the owner is correcting the draft.
  • Ledger inbox:
    • Done: #HD1C6W, #F3B2XC (cloZAPine exception confirmed by the owner), #VXB8XA, #E6142T (owner: won't fix).
    • Updated: #ZZ4RAP, #BFCTST.
    • New item for the missing WA valproate guideline.

RAG impact: behaviour change. Superseded (outdated) sources leave the answer evidence set, and the live corpus has none today. The guidance line has no retrieval effect and ships with an empty list. Canary pair: owner-approved, capped at US$4. The baseline is the 2026-10-21 14:09Z main canary (run 39614340500). The after run is post-merge.

Verification

  • Checks run:
    • tests/guidance-difference-note.dom.test.tsx: 7 passed.
    • Every unit test touching the answer payload or answer UI (36 files): Tests 845 passed (845).
    • rag-answer-fallback: 179 passed. Tier and loader tests: 23 passed.
    • npm run eval:rag:offline: 984 passed. npm run eval:rag:adversarial:offline: 25 passed.
    • tsc --noEmit clean, eslint clean, prettier applied.

Risk and rollout

  • Risk: low.
    • The outdated filter affects only documents marked outdated, and there are none live.
    • The guidance line is inert until an owner-signed entry exists.
  • Rollback: revert the individual commits.
  • Provider or production effects: none from this PR. Two read-only Supabase reads were made with owner approval.

Notes

  • Run npm run issues:reconcile after merge.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Jchk5iTkeUyPRUoCHyWei6