Eval job: each held-out case reviewed as a live Run is, eval.yml live, and a report that tells nothing a case expected (#309, PR 2)

Merged
DO
23d ago
+623-104

Part of #309. #309 closes once the Maintainer's first dispatch on main passes and its trial records the run by hashes alone; this is an open box on the issue.

This is #309's second pull request. It makes the eval job live: run gives each held-out case to the review Intent as a live Run is given its pull request, and .github/workflows/eval.yml runs the steps. Record 0078.

What changes

  • What is measured is what runs. Two refactors share the live Run's code with the eval job:

    • harness/task.py's build now reads the admission from the pass record and hands it to from_admission.
    • harness/run.py's job now calls answer.

    The eval job's run step (harness/eval/run.py) calls the same two functions. It gives each case's own admission, its facts and diff, no texts, no Findings, a fresh Run id, and the Release the dispatch names.

    • A test compares the Task the CLI receives against task.build for the same content.
    • It also checks that nothing of the home reaches the CLI: not the token, the owner or the repo.
    • A case whose facts don't name its diff's content builds no Task. It is not_this_set, and the model is not asked.
  • Time. The job runs on ubuntu-24.04 with a 120-minute timeout, both held by tools/check_workflows.py.

    • The time is counted from when plan was made, which it records, so the fetch and the install spend it too.
    • No case is started after 110 of the 120 minutes, nor once less time is left than the case's tier allows plus the Engine's two version reads.
    • A case not started is unanswered. A test ties the budget to the timeout.
  • The report always gets written. A case the review raises on is unanswered, and the report is written anyway, so judge fails the run.

    • An interrupt during a case, such as a cancel, asks nothing more: that case and every one after it are unanswered, and the report is still written.
    • A step that fails prints one code and nothing it raised.
  • The report says nothing about what a case expected (the Factory Builder's ruling of 2026-10-27). A case the scorer didn't pass is fail with the single code case_failed.

    • The scorer's own codes (finding_not_raised, criterion_not_met and the rest) never leave the step: not in the artifact, and not in the log.
    • eval-report/1 now refuses them, and a counter-example holds that.
    • The artifact is public, so those codes would assemble the answer key across dispatches.
  • The workflow. eval.yml runs these steps in order: checkout, plan, fetch, the CLI's install, run, the upload, judge.

    • The install is the Run's jobs' block, word for word, given the Engine's pin and its checksum.
    • The run step names the model's credential and the CLI's path, and nothing else.
    • check_workflows holds the install's place, its exact block and pin, the CLI path, the runner and the timeout, with 11 more planted faults.
  • Docs.

    • Record 0053's row that waited on harness/eval/run.py is re-pointed to its tests.
    • evals/AGENTS.md gains the three new rules.
    • harness/AGENTS.md no longer lists eval/run.py as unbuilt.
    • The workflows standard names eval.yml's install.

Still to do, after merge (#309's open box)

  • A dispatch on main reads the home and uploads a report of hashes, verdicts and codes alone.
  • A dispatch from another branch is refused by factory-eval's deployment branches.
  • A platform trial records both, with a fixture holding hashes only. It should also confirm that the deployment-branch policy is of type branch, so a tag named main does not match.
  • Each dispatch costs one review-tier call per case: 44 calls now that #363 has landed.

Accepted, noted in record 0078

  • Record 0053 names the home's owner, the Maintainer's public account. That opens nothing of a private repo. The repo's name stays secret, and HELDOUT_OWNER stays a secret so the pair is never printed together.
  • The vendor CLI now runs in the one job that holds both credentials, in separate steps. It is held by its sha256 and installed with the runner's passwordless sudo.

Found earlier, not changed here (from the security pass)

  • The CLI's environment doesn't set CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC, so its telemetry leaves the runner. Changing that would change live Runs too.

Checks

  • make verify passes.
  • Every new or changed mutant was applied by hand and its named tests run, and each is killed. CI's shards are the gate.
  • CodeQL: no alert expected. The checkout has no ref:, the trigger is workflow_dispatch, inputs.release reaches its step only through env:, and the actions are pinned.

Reviews

  • Claude review:
    • 1 major, fixed: the runner and timeout are now held, and the budget is tied to them.
    • 2 minors fixed: record 0078 now states the step order 0053 no longer holds, and evals/AGENTS.md gains the three new rules.
    • Nits fixed.
  • Security pass: no blocker or major.
    • Fixed: an interrupt printing its chained report; runs-on pinned exactly; the Task builder held to its two callers; the CLI added to the risk row.
    • Ruled: the report's codes.
    • Accepted: 0053's owner line.
  • Muse (Wonder Full), by the Maintainer's direction while Argon is out:
    • 1 major, fixed: an interrupt during a case left no report.
    • 3 minors, fixed: the budget now counts from the plan; the Release passed through the step and the fresh Run id are each held by a test and a mutant.
  • AGY: not run (quota). An Argon read of the merged diff is owed (CI/workflow and token handling).

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added a manually triggered evaluation workflow for running held-out review cases against a selected release.
    • Evaluation results are available as a downloadable report, retained for 90 days.
  • Improvements
    • Reports provide case verdicts and generic failure reasons without exposing case contents, model output, or detailed scorer failures.
    • Runs now produce a report when a case errors or is interrupted; incomplete cases are marked unanswered.