Eval report: the model that answered, the prompt and rubric by hash, and what each case used (#616)
Merged
DO
1d ago+835+835+012345678901234567890123456789-67-67-01234567890123456789
Closes #616, "The eval report records the model, prompt and rubric it ran, and each case's usage", of the gate-v1 set.
What changes
eval-report/1 now names what a run measured and what it cost:
| Field | Holds |
|---|---|
models | each model the CLI said answered a case, by its modelUsage id, not the configured name |
prompt_sha256, rubric_sha256 | the sha256 of roles/<role>/prompt.md and config/rubric-<role>.json at the commit checked out, or null where either is missing |
each case's usage | turns and cost_usd as the CLI reported them, seconds as the Harness timed the call; a failed Result's usage too; a case never asked used nothing |
- The judge's run line says
answered by X, not the configured Ywhen the answering model is not the configured one alone. - The gate (
verdict) is unchanged. The Engine already refuses another model's answer case by case (decision 0147). - A gate set's aggregate usage is left to #612, "The eval job reads a named set, and a gate set's report is aggregates only", as the Builder agreed.
Decision 0162 records it.
Checks
- One new test:
tests/test_evals.py::test_a_report_names_the_model_prompt_and_rubric_and_what_each_case_used. - Seven new counter-examples of the schema.
- 13 new mutants and 9 updated: all 176 mutants of
harness/eval/__init__.pyare killed at 8addd9ce, aftermainwas merged in. make verifypasses at 8addd9ce.- 04fe4e0a: the record's schema row names its test and eight schema mutants, one to each field, all killed (28 of 28 of the schema's), after
check_decisionsfound it named none in CI. - The merge keeps issue 610's
surplusbeside this PR'susages, surplus first; the mutants that name the report's call follow it.
Reviews
- Claude and Azure Astra read the branch. Their findings are fixed in c7971e1c and c730468f:
- a list given as a model id crashed
models(); - a failed Result's usage was untested;
- four schema counter-examples were missing;
- a fallback in the run line could never run.
- a list given as a model id crashed
š¤ Generated with Claude Code