#3579 eval: record why each answer source ranks where it does, opt-in on canary dispatch

User description

Summary

• Before: the answer-quality canary reports which files an answer used, but not why they landed in that order. After PR #3572 went live, assessment-documentation still lists Assessment Documentation (AKG) 4th behind Admissions, Radiotherapy and Pressure Injury. An offline reproduction keeps AKG first, so a step after selection on the live path is reordering, and nothing recorded shows which.
• After: the bounded per-source diagnostics also carry ranking (hybrid, rank, release-rank and final scores, final rank, and pipeline match reasons such as retrieval_signal:document_title_phrase or second_stage_rerank). A new dump_rag_cases dispatch payload passes the existing --dump-rag-cases flag, so the file lands in the canary artifact.
• How: buildBoundedRagSourceDiagnostics in scripts/eval-answer-quality.ts gains the ranking block. Reasons are bounded to 48 and passed through displaySafeDiagnosticText. eval-canary.yml adds DUMP_RAG_CASES to the answer step, which is empty for the weekly schedule. The dump is written before any --fail-on-threshold exit.

Areas touched: Answer engine
RAG impact: no retrieval behaviour change — eval diagnostics and an opt-in canary artifact only

Verification

• Checks run:
  • npx vitest run tests/eval-answer-quality.test.ts: Tests 23 passed (23), including the new ranking-diagnostics test.
  • npm run lint (changed files): recorded a pass for "lint:internal"
  • npm run typecheck: recorded a pass for "typecheck:internal"
  • npm run test: Tests 31524 passed | 8 skipped (31532)

Risk and rollout

• Risk: low. This is eval tooling only, and the workflow switch is off unless a dispatch sets dump_rag_cases: "true".
• Rollback: revert this commit.
• Provider or production effects: none from merging. A dispatched canary run with the switch on is paid (about US$1 to 2) and needs the owner's named yes.

Notes

• Kept as a draft until the owner says go.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VkKjYakMrMXofU8foQCgbq

Generated by Claude Code

Summary by cubic

Records each answer source's ranking scores and pipeline match reasons in the eval canary diagnostics, so a live source misorder (the AKG guideline still ranked 4th after PR #3572) can be diagnosed offline.

Diagnostics

• buildBoundedRagSourceDiagnostics now emits a ranking block: answer-evidence score, hybrid, rank, release-rank, and final scores, final rank, and match reasons (capped at 48 and sanitized).
• The answer-evidence score re-scores the emitted sources with the answer-stage ranking that set their order, since rankAnswerEvidence can reorder sources after retrieval selection; it explains that reorder, not later context-selection moves.
• eval-canary.yml adds a DUMP_RAG_CASES env var that passes --dump-rag-cases only when a dispatch sets dump_rag_cases: "true"; the weekly schedule stays off.
• Eval tooling only — no retrieval or answer behavior change.

CodeAnt-AI Description

Add opt-in source-ranking details to answer-quality canary dumps

What Changed

• Each answer source’s diagnostic details now show its ranking scores, final position, and reasons for matching.
• Dispatch runs can optionally save per-case answers and ranking details as a canary artifact; scheduled runs remain unchanged.

Impact

✅ Easier diagnosis of misordered answer sources
✅ Offline review of canary ranking decisions
✅ No extra answer generation for diagnostics

@codeant-ai ask: Your question here

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

@codeant-ai: Your feedback here

@codeant-ai: Do not flag unused imports.

@codeant-ai: review