Keep every matching table row per chunk (RAG-A-02, split from #3707)

Merged
BI
22d ago
+126-18

Split out of #3707 at the owner's request (21 Oct 2026), so that the table-row grouping gets its own canary pair.

  • RAG-A-02 Table facts are grouped per chunk (up to 5, deduplicated by id) instead of one. The chunk keeps its best fact's score, so scores and ordering are unchanged.

Before: when several table rows matched the same chunk, a lower-ranked row was dropped along with its lower score, so only about one structured row survived per chunk. After: every matching row is kept (the five highest-ranked by text_rank, ties in arrival order).

Changes: groupTableFactMatches and maxTableFactsPerChunk in src/lib/rag/rag-candidate-sources.ts, used by searchTableFactCandidates, plus tests/table-fact-grouping.test.ts.

RAG impact: behaviour change. This applies as soon as it deploys against the existing corpus (no re-index needed). Any query whose table RPC returns several rows for one chunk will now expose up to five rows instead of one.

Canary plan

The eval-canary only measures main (docs/rag-behaviour/safeguards.md), so the pair runs around the merge, and only once the OpenAI quota on the GitHub Actions key allows (the last run, 18 Oct, failed with "OpenAI quota is exhausted"):

  1. Baseline: dispatch eval-canary on main immediately before merging. It must be green.
  2. Merge this PR.
  3. Post-merge: dispatch eval-canary again and compare with --fail-on-regression (recall pinned at 1.0, zero per-case RR regressions).
  4. Any regression means a single-commit revert of this PR and a confirmation run.

Stays a draft until the quota is available and the baseline is green.

Evidence (local)

table-fact-grouping, table-facts and table-fact-ranking Vitest: 8 passed (b12d5c65f). npm run typecheck clean. eslint clean on the changed files.

Summary by cubic

Fixes table-fact grouping (RAG-A-02) so every matching table row per chunk is kept—up to five, deduplicated by id, keeping the highest-ranked facts—instead of only the single highest-ranked row. A higher-ranked row from a sibling variant displaces a lower-ranked one once the cap is full, and a duplicated fact keeps its higher-ranked copy.

The chunk's winning fact is now chosen by text rank rather than hybrid score, since the hybrid score is capped and can tie rows with high ts_rank_cd; ties keep the first row. The chunk's score remains tied to its best fact, so ranking and ordering are unchanged.

The change applies immediately on deploy with no re-index needed; any query returning multiple table rows for a chunk now exposes up to five rows instead of one. It adds groupTableFactMatches and maxTableFactsPerChunk in src/lib/rag/rag-candidate-sources.ts, with tests in tests/table-fact-grouping.test.ts.

Summary by CodeRabbit

  • Bug Fixes

    • Improved table-fact search results by grouping matches by source chunk.
    • Results now prioritize the highest-ranked facts, remove duplicate query variants, and retain up to five distinct facts per chunk.
    • Lower-ranked matches can be replaced when a stronger match is found, improving result quality and consistency.

  • Tests

    • Added coverage for ranking, deduplication, result limits, chunk separation, and score preservation.