CI: make verify runs in three legs, and a mutant its named tests miss is an error, not a whole-suite run

Merged
DO
9d ago
+323-36

On 2026-11-09 merge group pr-657-a81e6e58 was cut at the checks step's 26 minutes, and the queue ejected #657, a diff that only touched fixtures. In the same run mutants (7) finished at 26.2 of its 27. This PR gives both legs at least 40% headroom on merge-group runs. It skips and weakens nothing: every test, every seed and every mutant still runs in every merge group, and the required check is still verify. needs_passed is unchanged.

Protected edits (.github/workflows/verify.yml, tools/check_workflows.py): pre-approved by the Maintainer 2026-11-08, ruled by the Factory Builder 2026-11-09. Design: option (a) with three legs, plus the mutant fix and its guard. No decision record, following #180's precedent; no rule of AGENTS.md's table changes.

What was slow (17 runs to 2026-11-09)

LegLimitTookCause
checks, one job2614.1–23.4 min; cut at 26.2the whole suite in one process: test_tools 49%, test_simulator 40%, test_hooks 8%; checks 41 s
slowest mutants leg2713.8–26.1 mintwo mutants whose named test passes on them, so the leg ran the whole suite after it, for 10–21 min

The two mutants, found from the gaps in the job logs, with their killers found by running the whole suite on each:

  • "P-58 keeps the first head's status" (outlier legs in 5 of the last 6 runs): its named test passes; ReachScript.test_a_head_ends_where_the_next_line_is_no_head fails on it, in 1 s.
  • "the impostor workflow posts from the Lens's own file": its named test passes; Simulation.test_the_seeds_serve_items_with_an_acceptance_contract_and_without fails on it, in 5 s.

The change

  1. make verify in three legs. checks becomes a matrix, CHECKS_SHARD/CHECKS_SHARDS from strategy.job-index/job-total, as the mutant shards are set. Every leg runs every check. Its tests are every third test method of the whole suite, so across the legs every test runs once, all 300 seeds included. With neither variable set, make verify still runs everything, as at a desk. An unreadable leg (one variable without the other, not a number, or out of range) stops the run before anything else.
    Held by test_the_legs_of_make_verify_run_every_test_once (the shares partition the suite for 1, 2, 3 and 5 legs; the front door) and check_workflows._legs / test_the_legs_of_make_verify_are_told_their_place_and_count_by_the_matrix: the index and count come from the matrix and are never written in; the job's shape is closed, as the shards' is. The example and the standard's copy are changed to match.
  2. The two mutants name their killers.
  3. Guard: the mutant run (guarded → attempt(..., alone=True)) now runs a mutant's named tests alone. If they pass on it, the mutant is an ERROR, named_test_does_not_kill: … passed on it; name the test that fails on it, never a run of the whole suite. tools/untested.py still calls attempt without alone, so its edits still run the whole suite after their tests. Held by test_a_mutant_its_named_tests_miss_is_an_error_and_never_runs_the_whole_suite.

Each new rule has its mutant: 7 for _legs, 5 for the legs in tools/verify.py, and 4 for the guard. Two existing targets moved with the line they mutate. A filtered run of the 21 new and moved mutants killed all of them.

tools/AGENTS.md lost a line of prose, so that the guard's row fits its 40 lines.

Runner-minutes per run

  • Before: 134–164 over 15 jobs, median 143 (the last ten runs).
  • After: 140 over 17 jobs on this PR's run at 35673e9 (run 42565323152). The slowest checks leg took 9.1 of 26 (65% headroom) and the slowest mutants leg 14.4 of 27 (47%). The first runs dealt the legs unevenly: tests took 309, 355 and 675 s at 5524e9d, because round-robin put the slow simulator tests on two legs. The slowest leg there was still 11.9 of 26 (54%).

Verification

  • make verify on an export of 35673e9: 1,719 tests, OK (8 expected failures), exit 0.
  • python3 -m tools.mutants: CI's twelve shards, all green at 35673e9. On the first run, the guard caught a third misnamed mutant. "exactly on the bar does not clear it" now names test_each_kind_is_held_to_its_own_bar_where_the_cases_are, which holds the bar exactly.

Reviews

  • At 5524e9d: Argon clean (10 of 10 parts), Muse clean, Claude security review clean. The Claude review raised two minors, both fixed in 74753ff:
    • The impostor mutant's new killer failed only at the mutant run's 40 seeds. The test it named before now asserts, after every step, that the Lens's own App never posts from the Lens's own file. It kills the mutant at 5, 40 and 300 seeds, and the mutant names it again. This edits tests/test_simulator.py, a protected path, and only strengthens it: pre-approved by the Maintainer 2026-11-08, ruled by the Factory Builder 2026-11-09.
    • A named test that fails once and then passes was told as named_test_does_not_kill. It is SURVIVED again, as before the guard, with a mutant for the difference.
  • Later commits: b4b4c15 puts a test's injected env key into checks' own env:, since the legs gave it one. 35673e9 has the guard's catch above and a mutant target moved with the renamed loop.

After the merge, three consecutive merge-group runs will be reported, each with the slowest checks leg and the slowest mutants leg against their limits.

🤖 Generated with Claude Code