#25610 cmux NEXT Chief evals: pass rate per group, baseline diff, tone and delegation cases

Fixes #25575

What

The Chief fast-eval suite (Native/OptChat/optchat-chief/evals/) already met most of the issue: one headless command (evals/run.sh), JSON cases, and a CI workflow with workflow_dispatch. This PR adds the missing pieces:

• Pass rate per category. summary.json now holds group_pass_rate and case_results (each case's group, passes and runs; --repeat runs count as one case). Every run prints a table of pass rate per group.
• Diff against a baseline. chief_eval --baseline <dir|summary.json> (and run.sh --baseline ...) prints, and writes to diff.md and diff.json, each group's pass rate before and after, a total, the cases that regressed or improved, new and missing cases, and the groups left out of this run. The total counts only groups both runs have, so an --only run compares like for like. Older summaries that lack case_results diff by group only.
• CI on demand gets the diff. When you run chief-evals.yml by hand, it downloads the chief-evals artifact from the last feat-cmux-next run that has one, or from the run id you give as baseline_run. The diff appears in the job summary. The job needs actions: read to fetch it.
• Tone and delegation groups. The issue names both, and neither had its own group. Five new cases, bringing the total to 44:
  • tone-no-filler: no intro line, no "great question", no offer of more help.
  • tone-frustrated-no-groveling: work starts, and the reply has no run of apologies.
  • tone-no-self-narration: the reply gives the result, not "let me check".
  • delegation-long-work-to-subagent: build and test work goes to a subagent.
  • delegation-one-subagent-per-ask: one subagent per independent ask.
• Docs. The README gains "Adding a model case" and "Baseline diff". The design doc's groups table gains tone and delegation.

Defaults picked

• The baseline diff is a report, not a gate. The exit code still follows --min-pass (CI: CHIEF_EVAL_MIN_PASS, 0.8). Gating on regressions would need repeat runs to beat sampling noise, and the repo rule is not to add CI gates.
• The default CI baseline is the newest feat-cmux-next run that has a chief-evals artifact. Runs that skipped the model cases leave no artifact and are passed over.
• No CI steps or jobs were added: the baseline fetch lives inside the existing model-cases step.

Verification

• cargo test --example chief_eval on rbx: 3 unit tests of the summary, the diff and its Markdown (repeats folded, regressed/improved/new/removed, groups not run, an old-format baseline). All pass.
• rustfmt --check, and scripts/cmux-next/prepush-gate.py reports no new violations.
• Real model run on rbx: claude-opus-5-5 at medium effort, judge claude-sonnet-5, through the sr2 gateway. All 44 cases ran once: 41/44. The five new cases passed. The failures are in existing cases: assistant-a672-ack-silent (it posted something), assistant-a672-look-dont-ask (it asked a question) and automations-a675-laptop-asleep (94 words). Per group: acp 2/2, assistant 8/10, automations 2/3, delegation 2/2, links 7/7, memory 2/2, multiplayer 6/6, tenacity 6/6, tone 3/3, upsell 3/3.
• --only tone- --baseline <that run>: the diff table showed tone 3/3 → 2/3, the "groups in both" total and the "Groups not in this run" line, and it named tone-frustrated-no-groveling as regressed. That case is a real finding: across reruns the Chief replied in 77–79 words against the 70-word cap.

Observed / Not run

• Observed: the unit tests, the full model run, the --only + --baseline run and the prepush gate above.
• Not run: the workflow's new baseline download, which only runs on Actions. It is plain gh run list / gh run download, and a missing baseline is non-fatal. actionlint is not installed here; the YAML parses.
• The pre-commit hook also registered the existing tests/test_nightly_delivery_concurrency.py in tests/test-execution.toml. It was missing on base.

Changelog

none

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}