feat(eval): add a diff-impact task class and record its first baseline
Adds an eval task class for the diff-seeded explore (#125), which had none: the existing corpus holds only discovery questions. Its first run found that both arms name every other changed file in 34 of 36 sessions, so this set of cases does not yet separate the arms.
What it adds
scripts/agent-eval/diff-impact-cases.json: six real commits (four gin, two express).- A fixture builder: the commit's parent as a single-commit repo with one file edited and left uncommitted, so the answer cannot be read from history. The
.codegraph/index is excluded from the fixture's git status. - A scorer and summary: recall against the other source and test files the real commit changed. That truth set is a co-change proxy, so recall is the main score and precision is a floor.
run-diff-impact.sh: builds each fixture, indexes it, runs the existing with/without headless arms, scores them.- Six vitest tests for the scorer and fixture builder; a short section in the agent-eval skill.
docs/benchmarks/diff-impact-baseline-2026-10-11.md: the first run and its limits.
First run
36 sessions (6 cases x 2 arms x 3 repetitions, sonnet, effort high), about $5 and 13 minutes.
| Arm | Recall | Precision floor | Mean cost | Mean time |
|---|---|---|---|---|
| With CodeGraph | 97% | 25% | $0.150 | 19.1 s |
| Without | 97% | 29% | $0.139 | 19.2 s |
No clear difference. The arm with CodeGraph called codegraph_explore in 3 of 18 sessions; all 36 sessions ran git diff or git status and used grep, which is enough on repositories this size. The doc lists what to change next: drop the "see git diff" hint, add a larger repository, and compare a prompt that names the tool.
Checks
__tests__/diff-impact-harness.test.ts passes and npm run check:agent-docs passes. No product code changes.
Summary by cubic
Adds a diff-impact eval task class for the diff-seeded explore path (#125), which previously had no eval: the existing corpus only holds discovery questions. The first baseline found no clear difference between the arms—both named the other changed files in 34 of 36 sessions.
What it adds
- Six real commits (four gin, two express) as cases in
diff-impact-cases.json. - A fixture builder: the commit's parent becomes a single-commit repo with one file edited and left uncommitted, so history can't give away the answer;
.codegraph/never shows up in git status. - A scorer and summary: recall against the other source and test files the commit changed; precision is only a floor because the truth set is a co-change proxy. Files the commit added are dropped from the truth set, and
./-prefixed paths count as the same path. - A runner, six vitest tests, and a skill section. A failed arm no longer stops the loop: such runs are recorded unscored (null recall),
scores.jsonlresets per invocation, and stale run artifacts are cleared; the scripts resolve paths so the harness runs from any directory.
First run
- 36 sessions (6 cases × 2 arms × 3 repetitions, sonnet, effort high) at about $5; both arms scored 97% recall.
- The arm with CodeGraph called
codegraph_explorein 3 of 18 sessions; all sessions usedgit diff,git status, or grep, which suffices at this repo size. - The baseline doc lists next steps: drop the "see git diff" hint, add a larger repository, and compare a prompt that names the tool.
Summary by CodeRabbit
- Evaluation
- Added a repeatable benchmark for measuring how accurately coding assistants identify source and test files affected by an uncommitted change.
- Reports summarize recall and precision across evaluation setups and individual cases.
- Added six cases across Go and JavaScript repositories, with baseline results from 36 sessions. The baseline found similar recall with and without CodeGraph, and the results were inconclusive.
- Documentation
- Added guidance for preparing benchmark runs and interpreting results.