#3537 fix(rag,eval): Best Practice Prescription extractive-first top-up and blocking-gate retry

User description

Summary

• Best Practice Prescription extractive-first top-up (fix 1). The measured query "What does the best practice prescription document require?" already matches isSourceBoundBestPracticePrescriptionRequirementsQuery (class document_lookup, verified offline). What failed was the evidence: the fast four-chunk selection carried only the AKG procedure's p4 legislative/monitoring chunks (05a8819b, 212f78ad in canaries 38383361787 and 38419027058). So buildBestPracticePrescriptionRequirementsAnswer never saw its p1 program-use + p2 medication-profile pair (live chunks 321cfd4f, 3babd452; both pass the builder's existing regexes). The query fell through to gpt-5.6-terra, and the final gate rejected that answer as ungrounded_unsupported_answer. New src/lib/rag/rag-best-practice-prescription-evidence.ts mirrors the #3527 admission/discharge top-up. For this one query only, and only when the selection cannot form the pair, it appends at most one governed eligible chunk per role from the same document, using the builder's own predicates (now exported, otherwise unchanged). The short-circuit still requires the candidate to pass every final quality gate. The answer text, regexes and source content are not changed.
• Blocking-gate retry (fix 3). Every canary red since 2026-09-13 was a different single-case gate: citation_failure_rate 0.0227 (one case in 44), one unaccounted review fallback, and one 6.4 s refusal pushing route-unsupported p95 to 6360 against 4000. eval:quality --fail-on-threshold now re-runs only the cases implicated in a blocking RAG failure, once (scripts/lib/eval-quality-retry.ts). It then rebuilds the report with each retry substituted for its first attempt, so a gate blocks only if the same case fails again. Latency gates are re-evaluated as p95 over the population that includes the retry samples. Retrieval and offline-invariant failures are never retried. Every threshold value is unchanged (pinned by a test). The report gains a ## Blocking-Gate Retry section and a gate_retry JSON field listing first-attempt blocking failures, the retried cases with first/retry failures, latency and routing reason, and the blocking failures after retry. --no-gate-retry keeps the old behaviour.
• Not in this PR: assessment-documentation ranking (fix 2). Offline replay of the live chunks through rankAnswerEvidence and selectModelContextEvidence, with equal retrieval scores, already ranks Assessment Documentation (AKG) 1–3. Live, only its weaker p3 chunks reach selection, so the loss is upstream in candidate retrieval/scoring. I could not observe that without the live search diagnostics, and the read-only text RPC returned no rows outside the app context. Shipping an unmeasured ranking change is exactly what docs/rag-behaviour/refuted-approaches.md forbids, so this needs a scoped follow-up with live score explanations. The fixture is unchanged.
• Owner approval: BigSimmo approved all three fixes on Oct 13 2026 (11:32 PM AWST), including the protected-surface change and canary-pair process.

Areas touched: Answer engine (plus 1 file not yet placed in any area)

RAG impact: behaviour change — canary pair 38419027058 -> planned post-merge

Verification

• Checks run: npx vitest run tests/rag-best-practice-prescription-evidence.test.ts tests/rag-extractive-first.test.ts tests/rag-admission-discharge-comparison-fallback.test.ts gives 34 passed. The first new test reproduces the live miss (the p4-only selection does not short-circuit). npx vitest run tests/eval-quality-retry.test.ts tests/eval-quality.test.ts gives 49 passed. tests/eval-canary-workflow.test.ts + tests/rag-imputation-contract.test.ts give 16 passed. eslint is clean on the changed files, tsc --noEmit reports no errors in the changed files, and the maintainability budgets pass (rag-extractive-answer.ts 5234/5269).

Verification not run: verify:pr-local and provider-backed eval:quality / eval:retrieval:quality. The box was overloaded so only narrow suites ran, and provider evals are owner-run. Under the canary-pair protocol the eval-canary measures only main. Baseline is the latest canary on main, 38419027058 (red; last green 34122919100 on 1d6ba4fa). The post run is one owner-approved eval-canary dispatch after merge, once main CI settles. Pass criteria: best-practice-prescribing routes validated_best_practice_prescription_requirements_extractive_first and grounds; golden retrieval stays 36/36 with zero per-case rr regressions.

Risk and rollout

• Risk: low. Fix 1 can only append evidence for one exact query string and still fails closed through the unchanged final gates. Fix 3 can turn a red canary green only when the implicated case passes on its retry. It can also add provider cost for the retried cases (typically 1–3 cases).
• Rollback: revert either commit independently. They are separate (fix(rag) and fix(eval)).
• Provider or production effects: None at merge. Retries run only inside provider-backed eval:quality --fail-on-threshold runs.

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use
• No patient-identifiable document workflow was introduced or expanded without explicit governance approval
• Supabase target remains PsychSift Production (sjrfecxgysukkwxsowpy)
• Service-role keys and private document access remain server-only
• Demo/synthetic content remains clearly separated from real clinical sources
• Source metadata, review status, and outdated/unknown-source behavior remain conservative
• Deployment classification/TGA SaMD impact was checked when clinical decision-support behavior changed

The last item is left unchecked deliberately. Fix 1 changes which already-shipped deterministic answer serves one administrative procedure question, and no SaMD review was done for this PR. No clinical content, answer wording or source text changed. The top-up chunks pass the same governed context-pack admission as the packer.

Notes

• Investigation: canary runs 34773883150, 38363682098, 38383361787, 38419027058. Live chunk text was read with read-only SQL.
• Auto-merge is not armed: this is a RAG-ranking/clinical-risk PR with a canary-pair requirement, not an ordinary PR.

Summary by cubic

Fixes the Best Practice Prescription "requirements" query falling through to model generation when the fast four-chunk selection carried only the procedure's legislative/monitoring chunks. When the selected evidence can't form the program-use + medication-profile pair, the extractive-first builder now appends at most one governed eligible chunk per role from the same document, using the builder's own (now exported) evidence predicates. The deterministic answer fires as before, and the candidate still passes every final quality gate; answer text, regexes, and source content are unchanged.

Fixes eval:quality --fail-on-threshold blocking on single-case RAG gates. Every canary red since 2026-09-13 was a different one-case gate (one citation miss in 44, one unaccounted review fallback, one slow refusal pushing route p95 over the limit). The command re-runs only the cases behind a blocking RAG failure once, substitutes each retry for its first attempt, and gates only when the same case fails again. Retry selection shares the grounded-supported predicate with the gates.

• Latency and route-ceiling gates never retry (a retry can't clear them) and always block; retrieval and offline-invariant failures also never retry.
• First attempts stay in latency p95 populations, route buckets, estimated cost, and route-ceiling counts, so a retried case can't hide a slow or over-ceiling first attempt.
• A rerun that throws keeps the whole first-attempt verdict, with each gate listed once at its first-attempt value; partial passing retries can never lift a rate gate.
• Threshold values are unchanged and pinned by tests. --no-gate-retry keeps the old behavior. Diagnostic dumps tag attempts as initial or retry; the report gains a ## Blocking-Gate Retry section and gate_retry JSON field.

Rollout and scope

• No production behavior changes at merge: the top-up applies only to the one measured query, and retries run only inside provider-backed eval:quality runs.
• The two fixes are separate commits (fix(rag) and fix(eval)), so each can be reverted independently.
• Also pins the clock in the roster give-away test, which had started failing on every run from 14 Oct 2026 because the fixture shift counts as urgent inside 24 hours.
• The Assessment Documentation (AKG) ranking loss is a separate upstream retrieval-scoring issue and needs a scoped follow-up; the fixture is unchanged.

CodeAnt-AI Description

Restore grounded Best Practice Prescription answers and retry failed quality checks

What Changed

• Best Practice Prescription questions can now use the required program-use and medication-profile evidence when the initial results miss it, allowing a grounded answer instead of a model-generated answer that may be rejected.
• Blocking quality checks rerun only the answer cases linked to a failure, once. Repeated failures still block, and latency checks continue to count the original attempt.
• Reports show retry outcomes; if a retry fails to run, the original evaluation verdict is kept.

Impact

✅ Grounded Best Practice Prescription answers
✅ Fewer one-off quality-check blocks
✅ Slow first attempts still count toward latency limits

@codeant-ai ask: Your question here

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

@codeant-ai: Your feedback here

@codeant-ai: Do not flag unused imports.

@codeant-ai: review