Evals: the review Intent's held-out set has its 44 cases, and the cutover's rule is held (#183)
Merged
DO
23d ago+188+188+012345678901234567890123456789-16-16-01234567890123456789
Closes #183. The last Done-when of "The review Intent's held-out eval set, from the proof of concept's merged-then-reverted pull requests": the cases exist, and the rule that waited on them is held.
What it holds
evals/pr-review/cases.json:count44, and each case's id and the sha256 of its file in the private home. The ids are random and listed sorted, so neither an id nor its place says what a case expects. Nothing else of any case is here. The set meets #183's floor of 20 cases of each kind.- The cutover's rule stops waiting.
test_the_review_intents_eval_set_has_its_casesloses@pending, which it would otherwise fail as an unexpected success. Its rows inevals/AGENTS.mdand decision 0010 lose "not held yet". test_a_set_of_no_case_ends_the_run_before_anything_is_readwrites its own empty manifest instead of reading this repo's. It read the real manifest and asserted it was empty, which no longer holds. Ruling 8's behaviour is tested unchanged.
How the home was checked
- Each of the 44 files validates as
eval-case/1. Each is under the port's 1 MiB limit. - Each scores
passwithtools/eval_result.pygiven a Result that answers it rightly, andfailgiven one that answers it wrongly. None scoresunreadable. - Each was read back through
GET /repos/{owner}/{repo}/contents/pr-review/{id}.json, the pathadapters/heldout.pyreads, and hashed to the sha256 here, 44 of 44. - How the cases were chosen is written in the home's README, not here.
Gates
python3 -m tools.verify: 1275 tests OK (12 expected failures), exit 0.maingives the same at a3299a3.- An earlier
make verifyrun on this change exited 1. Its output was cut off, and the full rerun above passed, so the cause is unknown. Possibly a wall-clock test under load (not confirmed).
🤖 Generated with Claude Code
Summary by CodeRabbit
- Documentation
- Clarified the evaluation-set guidance in the project’s decision record and review instructions.
- Tests
- Added 44 review evaluation cases.
- Added validation that an empty evaluation set is reported and does not create temporary files.