#628 The eval job reads a named set, and a gate set's report is counts alone, its ledger sealed (#612, record 0164)

Issue #612, "The eval job reads a named set, and a gate set's report is aggregates only". It follows the Maintainer's ruling: seal now, decide storage later.

The ciphertext lives only as a run artifact under its retention until a storage decision is made. This PR adds no ledger job, no factory-ledger environment, no token and no write to another repo. No automated process reads a ledger, and the gate's bars apply to summed case-runs (#613's condition).

What changes

• Named set. eval.yml takes an input set, defaulting to pr-review. Only the plan step receives it, as EVAL_SET.

  • The plan reads evals/<set>/cases.json. The name must be a single path part; anything else is eval_set_unreadable.
  • The plan keeps the name, and every later step reads that set. The Release's gate still reads pr-review.

• Gate set. eval-set/1 gains gate, and each case gains natural. A gate set must mark every case. Nothing passed to the Role carries the mark.

• eval-aggregate/1, a gate set's report. It holds counts only:

  • passes;
  • per kind: cases, passes and a Wilson 95% interval;
  • natural and constructed cases counted apart;
  • codes of unread cases, each with its count;
  • usage, summed;
  • the Engine pin and the sha256 of the prompt and the rubric, and the models that answered (#616's models);
  • the bars applied, plus split_ok, kinds_pass and blocking_pass;
  • the ledger's salted sha256 and its ciphertext's sha256.

  It holds no case id, per-case verdict or per-case code.

• eval-ledger/1. Each case as eval-report/1 lists it (usage among it, since #625), with its natural mark and whole Result. It is written to the scratch directory with mode 0600, under a salt drawn for the run.

• harness.eval.seal runs after the run step and is given nothing. It:

  • checks each certificate against the sha256 pinned under .github/ (a protected path), and copies the checked bytes;
  • encrypts with openssl cms -encrypt -aes256, using -keyopt rsa_padding_mode:oaep for each recipient;
  • reads the envelope back and refuses unless every recipient is RSA-OAEP and the content is AES-256;
  • writes the ciphertext's sha256 into the report and removes the plaintext.

  A gate set's report without that hash is unreadable to the gate. A development set's run seals nothing.

• The upload. The ciphertext alone goes up as eval-ledger (90 days, if-no-files-found: ignore), before the report.

  • The seal step (id: seal) sets the job's one output, ledger_sha256. The report carries the same hash.
  • A file or link already at eval-ledger.cms fails the run for either kind of set (ledger_in_workspace). openssl seals a checked copy of the ledger into scratch, and only an envelope that reads back whole is copied into the workspace (O_EXCL|O_NOFOLLOW).
  • OAEP uses SHA-256 with MGF1-SHA256, and the read-back checks it.
  • The run step's budget and timeout drop from 110 to 109 minutes to give the seal its minute, so the job's limits still leave 2 minutes free.

• Guards on gate sets.

  • The plan refuses a set that is not a gate set but names a gate set's case by id or hash (eval_set_overlaps_a_gate_set).
  • A gate set may not mark a natural or constructed group of 1 to 9 cases.
  • Not built here: dropping gate from a gate set's own manifest is still only caught by review, because no hook protects evals/. Protecting it is a change to config/hooks.json, which is yours to decide.
  • The Release gate reads the commit's latest successful eval dispatch, so pr-review must be dispatched last before a cut. Another set dispatched later closes the gate. The record and the cut skill say so.

• verdict. A gate set's report goes to aggregate_verdict. It checks that the counts add up to the set, to the manifest's natural cases and to the report's own words. It also checks that the interval matches the counts, that the bars equal the job's (bars_differ otherwise), and that each kind clears its bar.

  • A report of the wrong kind for its set is report_of_another_kind.
  • A development set's report, eval-report/1, is unchanged.

• Rules. They are in a new harness/eval/AGENTS.md, because evals/AGENTS.md and harness/AGENTS.md are both at the 40-line cap.

• Record 0164. It amends decisions 0099 and 0150 for gate sets only. Their rows keep a count of a kind out of the report; a gate set's report carries one, counted where the cases are and held by verdict.

With no certificate yet

Until the Maintainer makes his key pair and commits a certificate and its pin, no pin file exists. A gate set's run then stops at the seal with no_recipient:

• Nothing after the seal runs, so neither the ledger nor the report is uploaded.
• The plaintext stays in the scratch of a runner that is thrown away.
• It is never uploaded in the clear, and never sealed to nobody.

The record says the same. Development sets, including pr-review today, are untouched.

Checks

Check	Result
SIM_SEEDS=40 make verify	OK, 1677 tests (at 857f45ab, the record numbered 0164)
Filtered mutants: every one on harness/eval/, eval.yml and eval.yml's shape (52 new, 16 brought up to date)	All killed (258 in the full run, then each changed one again)
Filtered mutants after merging #625: every one on the gate report and #616's report fields (27)	All killed
gitleaks	no leaks, 4 commits
Argon (8d7759e1)	5 major. Fixed: the leftover ciphertext and the within() path (one fix), split_ok when no kind was read, and the ledger under codes-only. Not changed: "blocking_pass vs kinds_pass['blocking']", because kinds_pass has no such key
Muse	not run (quota), owed after 2026-11-09
Astra	not run (quota), owed after 2026-11-09
Claude review (8d7759e1)	1 high and 2 medium fixed or stated in the record; 4 low fixed
Claude security (8d7759e1)	1 medium and 3 low fixed; the residual (dropping gate) is named above

🤖 Generated with Claude Code