Eval job: each held-out case reviewed as a live Run is, eval.yml live, and a report that tells nothing a case expected (#309, PR 2)
Part of #309. #309 closes once the Maintainer's first dispatch on main passes and its trial records the run by hashes alone; this is an open box on the issue.
This is #309's second pull request. It makes the eval job live: run gives each held-out case to the review Intent as a live Run is given its pull request, and .github/workflows/eval.yml runs the steps. Record 0078.
What changes
-
What is measured is what runs. Two refactors share the live Run's code with the eval job:
harness/task.py'sbuildnow reads the admission from the pass record and hands it tofrom_admission.harness/run.py'sjobnow callsanswer.
The eval job's
runstep (harness/eval/run.py) calls the same two functions. It gives each case's own admission, its facts and diff, no texts, no Findings, a fresh Run id, and the Release the dispatch names.- A test compares the Task the CLI receives against
task.buildfor the same content. - It also checks that nothing of the home reaches the CLI: not the token, the owner or the repo.
- A case whose facts don't name its diff's content builds no Task. It is
not_this_set, and the model is not asked.
-
Time. The job runs on
ubuntu-24.04with a 120-minute timeout, both held bytools/check_workflows.py.- The time is counted from when
planwas made, which it records, so the fetch and the install spend it too. - No case is started after 110 of the 120 minutes, nor once less time is left than the case's tier allows plus the Engine's two version reads.
- A case not started is
unanswered. A test ties the budget to the timeout.
- The time is counted from when
-
The report always gets written. A case the review raises on is
unanswered, and the report is written anyway, sojudgefails the run.- An interrupt during a case, such as a cancel, asks nothing more: that case and every one after it are
unanswered, and the report is still written. - A step that fails prints one code and nothing it raised.
- An interrupt during a case, such as a cancel, asks nothing more: that case and every one after it are
-
The report says nothing about what a case expected (the Factory Builder's ruling of 2026-10-27). A case the scorer didn't pass is
failwith the single codecase_failed.- The scorer's own codes (
finding_not_raised,criterion_not_metand the rest) never leave the step: not in the artifact, and not in the log. eval-report/1now refuses them, and a counter-example holds that.- The artifact is public, so those codes would assemble the answer key across dispatches.
- The scorer's own codes (
-
The workflow.
eval.ymlruns these steps in order: checkout,plan,fetch, the CLI's install,run, the upload,judge.- The install is the Run's jobs' block, word for word, given the Engine's pin and its checksum.
- The
runstep names the model's credential and the CLI's path, and nothing else. check_workflowsholds the install's place, its exact block and pin, the CLI path, the runner and the timeout, with 11 more planted faults.
-
Docs.
- Record 0053's row that waited on
harness/eval/run.pyis re-pointed to its tests. evals/AGENTS.mdgains the three new rules.harness/AGENTS.mdno longer listseval/run.pyas unbuilt.- The workflows standard names eval.yml's install.
- Record 0053's row that waited on
Still to do, after merge (#309's open box)
- A dispatch on
mainreads the home and uploads a report of hashes, verdicts and codes alone. - A dispatch from another branch is refused by
factory-eval's deployment branches. - A platform trial records both, with a fixture holding hashes only. It should also confirm that the deployment-branch policy is of type
branch, so a tag namedmaindoes not match. - Each dispatch costs one review-tier call per case: 44 calls now that #363 has landed.
Accepted, noted in record 0078
- Record 0053 names the home's owner, the Maintainer's public account. That opens nothing of a private repo. The repo's name stays secret, and
HELDOUT_OWNERstays a secret so the pair is never printed together. - The vendor CLI now runs in the one job that holds both credentials, in separate steps. It is held by its sha256 and installed with the runner's passwordless
sudo.
Found earlier, not changed here (from the security pass)
- The CLI's environment doesn't set
CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC, so its telemetry leaves the runner. Changing that would change live Runs too.
Checks
make verifypasses.- Every new or changed mutant was applied by hand and its named tests run, and each is killed. CI's shards are the gate.
- CodeQL: no alert expected. The checkout has no
ref:, the trigger isworkflow_dispatch,inputs.releasereaches its step only throughenv:, and the actions are pinned.
Reviews
- Claude review:
- 1 major, fixed: the runner and timeout are now held, and the budget is tied to them.
- 2 minors fixed: record 0078 now states the step order 0053 no longer holds, and
evals/AGENTS.mdgains the three new rules. - Nits fixed.
- Security pass: no blocker or major.
- Fixed: an interrupt printing its chained report;
runs-onpinned exactly; the Task builder held to its two callers; the CLI added to the risk row. - Ruled: the report's codes.
- Accepted: 0053's owner line.
- Fixed: an interrupt printing its chained report;
- Muse (Wonder Full), by the Maintainer's direction while Argon is out:
- 1 major, fixed: an interrupt during a case left no report.
- 3 minors, fixed: the budget now counts from the plan; the Release passed through the step and the fresh Run id are each held by a test and a mutant.
- AGY: not run (quota). An Argon read of the merged diff is owed (CI/workflow and token handling).
🤖 Generated with Claude Code
Summary by CodeRabbit
- New Features
- Added a manually triggered evaluation workflow for running held-out review cases against a selected release.
- Evaluation results are available as a downloadable report, retained for 90 days.
- Improvements
- Reports provide case verdicts and generic failure reasons without exposing case contents, model output, or detailed scorer failures.
- Runs now produce a report when a case errors or is interrupted; incomplete cases are marked unanswered.