fix(retrieval): rank a document first when a lookup names its whole title

Merged
BI
1mo ago
+239-1

Summary

Before: "What assessment documentation is required?" retrieved Assessment Documentation (AKG) but ranked it 4th. Admissions Policy, Radiotherapy Skin Care Assessment and Pressure Injury Prevention filled the top three because they share "assessment" or "documentation". The answer opened by quoting the ICU pressure injury policy. This is the answer-quality case that has failed since 7 Oct.

After: on a document lookup, a document whose whole title the question names is lifted above documents that share only one word with it.

How: rankClinicalResults already puts the AKG document first (its rank score is about 0.6 higher). But selectRetrievalEvidence then re-sorts on the clamped raw keyword score, and the other three chunks had a higher raw score because their content contains the literal phrase. That undid the clinical ranking. This PR adds a document_title_phrase selection signal for document_lookup queries. It fires when the core title (title or file name, without the extension or a known bracketed publisher code, at least two words) appears as a phrase in the query, and it adds +0.12 inside the existing clamp. Population qualifiers such as (Adult) or (CAMHS) stay part of the title, and a version or year in the query must also be on the title. It changes no comparator key, adds nothing above the relevance score and leaves the clamped-score contract intact. Saturated candidates are unaffected.

Areas touched: Answer engine
RAG impact: behaviour change — canary pair 38873736787 -> 38888159940

Verification

  • New test fails without the fix and passes with it.
  • npm run test:focused on the change: Tests 1730 passed (1730)
  • npm run eval:rag:offline: Offline RAG fixture and production-contract checks passed.
  • eslint clean. typecheck:internal pass.
  • Live eval-canary pair, approved by the owner. Baseline 38873736787, post 38888159940 on main b724bb545. --fail-on-regression compare: zero per-case reciprocal-rank regressions, document and content recall 1.0, mrr@10 0.9106 (+0.0185), content mrr 0.9475 (+0.0069), ndcg 0.9350 (+0.0067), failed cases 0. No revert needed.

Follow-up

  • The answer-quality case assessment-documentation still fails after this change: the answer path's sources still list Admissions, Radiotherapy and Pressure Injury above the AKG document, the same order as before. An offline reproduction with the selection fixture and the second-stage rerank keeps AKG first, so a step after selection on the live answer path (or live candidate scores unlike the fixture) is still reordering. Diagnosis needs live per-step telemetry and is tracked separately.

Risk and rollout

  • Risk: a protected ranking surface (src/lib/retrieval-selection.ts). The canary pair above is the gate and came back clean.
  • Rollback: revert the squash commit b724bb545.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VkKjYakMrMXofU8foQCgbq