Eval report: the model that answered, the prompt and rubric by hash, and what each case used (#616)

Merged
DO
1d ago
+835-67

Closes #616, "The eval report records the model, prompt and rubric it ran, and each case's usage", of the gate-v1 set.

What changes

eval-report/1 now names what a run measured and what it cost:

FieldHolds
modelseach model the CLI said answered a case, by its modelUsage id, not the configured name
prompt_sha256, rubric_sha256the sha256 of roles/<role>/prompt.md and config/rubric-<role>.json at the commit checked out, or null where either is missing
each case's usageturns and cost_usd as the CLI reported them, seconds as the Harness timed the call; a failed Result's usage too; a case never asked used nothing
  • The judge's run line says answered by X, not the configured Y when the answering model is not the configured one alone.
  • The gate (verdict) is unchanged. The Engine already refuses another model's answer case by case (decision 0147).
  • A gate set's aggregate usage is left to #612, "The eval job reads a named set, and a gate set's report is aggregates only", as the Builder agreed.

Decision 0162 records it.

Checks

  • One new test: tests/test_evals.py::test_a_report_names_the_model_prompt_and_rubric_and_what_each_case_used.
  • Seven new counter-examples of the schema.
  • 13 new mutants and 9 updated: all 176 mutants of harness/eval/__init__.py are killed at 8addd9ce, after main was merged in.
  • make verify passes at 8addd9ce.
  • 04fe4e0a: the record's schema row names its test and eight schema mutants, one to each field, all killed (28 of 28 of the schema's), after check_decisions found it named none in CI.
  • The merge keeps issue 610's surplus beside this PR's usages, surplus first; the mutants that name the report's call follow it.

Reviews

  • Claude and Azure Astra read the branch. Their findings are fixed in c7971e1c and c730468f:
    • a list given as a model id crashed models();
    • a failed Result's usage was untested;
    • four schema counter-examples were missing;
    • a fallback in the run line could never run.

šŸ¤– Generated with Claude Code