eval: gate retrieval changes by replaying saved agent queries
A change to existing retrieval (explore ranking, budgets, rendering, search) can now be checked in about 10 seconds by replaying 46 real agent queries against the old and new build, instead of a 15–40 minute agent A/B.
The change-gating agent A/Bs run so far mostly agreed with the deterministic probes. The defects only agents caught came from query shapes the probes did not have, and replaying those agents' queries reproduced each one. This keeps the query shapes and drops the per-change agent run.
scripts/agent-eval/replay-queries.mjs <checkout> --tasks <corpus.json> --base <build> --cand <build> --out <dir>answers every query with each build throughprobe-explore.mjs(passing each query's ownmaxFiles), prints a table of answers whose text differs, and exits 1 when an answer lost its Source Code section, a file the base rendered (marked when the query names the file) or amustMatchpattern. It refuses a checkout that is not at the corpus's pinned commit.scripts/agent-eval/harvest-queries.mjs --public <owner/repo> <transcript dirs>collectscodegraph_explorequeries from Claude Code transcripts and Codex rollouts (MCP calls and code-mode calls). It keeps only queries whose project'soriginis on github.com and named with--public, keeps each call'smaxFiles, and writes no paths.scripts/agent-eval/replay/codegraph.json: 46 queries agents asked about this repository, pinned to2a4af7c3, each read by hand.- Policy:
AGENTS.mdanddocs/AGENTS.mdnow gate a change to existing retrieval by the replay, with every changed answer explained in the PR. The agent A/B stays for a new language or framework, and runs once per release batch to add new query shapes to the corpus.
Replay on codegraph.json (46 queries, this repository at 2a4af7c3) | Identical | Changed, nothing lost | Flagged | Time |
|---|---|---|---|---|
| Same build twice | 46 | 0 | 0 | 9 s |
| Build whose default limit is two files (injected regression) | 30 | 0 | 16 (4 lost a file the query named) | 9 s |
#500 (root __tests__/ counts as a test), before vs after | 40 | 6 | 0 | 9 s |
29 of the 46 queries replay with the file limit (maxFiles) the agent asked for, so the injected default limit only reaches the other 17; it flagged 16 of them. One of the six #500 answers changed text without changing length, which is why the replay compares the saved answer text, not sizes.
Checks (at 591d5742): __tests__/query-replay.test.ts (new, 9 tests) covers the comparison rules (including equal-length text changes), all three transcript shapes (including braces and single quotes inside code-mode strings, and maxFiles), exact-host GitHub remotes, and fails a committed corpus that contains a local path; the 4 tests added for review findings fail on the first commit's scripts. An --ids filter that selects nothing exits 2. Full suite: 754 files, 9,308 passed, 40 skipped (vitest run, Node 24.22.0). npm run check:agent-docs: AGENTS.md 13,397 of 32,768 bytes.
Not covered: corpora for other repositories. The harvest also found 20 queries on a public fork of t3code and 2 on house-rules; they need pinned checkouts and are left for a later change.
README: checked the "About this fork" section; it describes no evaluation procedure and quotes no number this changes. No user-visible feature, command or config key changes, so there is no CHANGELOG entry.