Evals: the review Intent's held-out set has its 44 cases, and the cutover's rule is held (#183)

Merged
DO
23d ago
+188-16

Closes #183. The last Done-when of "The review Intent's held-out eval set, from the proof of concept's merged-then-reverted pull requests": the cases exist, and the rule that waited on them is held.

What it holds

  • evals/pr-review/cases.json: count 44, and each case's id and the sha256 of its file in the private home. The ids are random and listed sorted, so neither an id nor its place says what a case expects. Nothing else of any case is here. The set meets #183's floor of 20 cases of each kind.
  • The cutover's rule stops waiting. test_the_review_intents_eval_set_has_its_cases loses @pending, which it would otherwise fail as an unexpected success. Its rows in evals/AGENTS.md and decision 0010 lose "not held yet".
  • test_a_set_of_no_case_ends_the_run_before_anything_is_read writes its own empty manifest instead of reading this repo's. It read the real manifest and asserted it was empty, which no longer holds. Ruling 8's behaviour is tested unchanged.

How the home was checked

  • Each of the 44 files validates as eval-case/1. Each is under the port's 1 MiB limit.
  • Each scores pass with tools/eval_result.py given a Result that answers it rightly, and fail given one that answers it wrongly. None scores unreadable.
  • Each was read back through GET /repos/{owner}/{repo}/contents/pr-review/{id}.json, the path adapters/heldout.py reads, and hashed to the sha256 here, 44 of 44.
  • How the cases were chosen is written in the home's README, not here.

Gates

  • python3 -m tools.verify: 1275 tests OK (12 expected failures), exit 0. main gives the same at a3299a3.
  • An earlier make verify run on this change exited 1. Its output was cut off, and the full rerun above passed, so the cause is unknown. Possibly a wall-clock test under load (not confirmed).

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Documentation
    • Clarified the evaluation-set guidance in the project’s decision record and review instructions.
  • Tests
    • Added 44 review evaluation cases.
    • Added validation that an empty evaluation set is reported and does not create temporary files.