PR #634 - combsmith/software-factory

This adds the advisory check of issue 589, "A changed line of Python that no test notices fails verify, unless the change says it keeps behaviour". It follows the design note and the Factory Builder's rulings of 2026-11-07. It fails no pull request: for a month from its merge it reports, and this lane posts a weekly list to the issue.

What the Maintainer was asked

.github/workflows/untested.yml is a new file under a protected path. The Maintainer approved it on 2026-11-08 ("Approve (Recommended)"), running on both pull_request and merge_group.
Nothing is made required. The switch stays open until the month ends (ruling 3). The record keeps both routes, verify's needs or a second ruleset context, and the trade-off of each against the emergency fix (design §9.1).
This PR's own untested run will fail. The job runs the base's tools/untested.py, and the base has none yet. For the same reason this PR's merge-group run fails once too: it checks out merge_group.base_sha, which is main without the tool. It is in no job's needs (verify needs checks, commits and mutants), and main requires only verify, so it blocks nothing. Every later PR runs it, and every later merge group declines it.
A one-time exception at enqueue. enqueue_clean.sh waits for every check to be green, so this PR can't pass it. With the Factory Builder's go-ahead of 2026-11-08, it was enqueued directly with enqueuePullRequest at its reviewed head, once every other check and the merged-tree verify were green.

What it does

The job checks out the base and runs python3 -m tools.untested. The tool:

fetches the head and exports it with git archive;
reads each hunk of git diff -U0 from the merge base, in a *.py outside tests/, trials/, examples/ and schemas/, leaving out the entries of the ISSUE and MUTANTS lists of tools/mutants.py, but not the code among them;
skips a hunk whose module reads the same to Python without it (docstrings, comments, and annotations under from __future__ import annotations);
reverts each remaining hunk alone, and runs the head's tests on it through tools/mutants.py's own attempt, so a kill is confirmed by two runs of the one test that failed.

Case	Outcome
A test notices the revert, by an assertion or an exception outside `CRASH`	killed
No test notices	SURVIVED, with a warning on its lines and the trailer line it would need
The edit does not parse, its module will not import, or only a `SyntaxError`, `ImportError`, `NameError` (and kin) or a test module that will not import notices it	Not a kill. It falls back to the file's every hunk reverted together, whose kill is counted apart (`killed_by_file`), then, for a hunk adding more than 40 lines alone, to at most 6 mutants of those lines (ruling 1). A mutant that hangs makes the hunk an ERROR; one the budget left makes it UNCHECKED
No edit of the hunk can be run, or the 20-minute budget ran out first	UNCHECKED, with its reason. No run, import probe or read of the suite begins past the budget but a kill's confirmation, and none lasts past one `TIMEOUT` after it, so the job ends inside its step
Two failures that are not the edit's	FLAKY

The tests run first:

For a hunk in tools/: each test class that reaches the module (ruling 2), and for a class in WHOLE_TOO_LONG, its methods.
Elsewhere: each test module that imports the module at any remove.
The slow simulator modules are left to the whole suite, which runs only when the first tests pass.

The trailer. Behaviour-preserving: <path>:<start>-<end>, read from the head commit by git's own parser, covers a survivor or an UNCHECKED hunk when it holds every line the hunk added. A trailer naming no changed line, or a file the change leaves alone, is a problem.

Advisory, by construction:

untested.yml runs on pull_request and merge_group. check_workflows requires both of any workflow on pull_request, so on a merge group the tool says it did not run, and exits.
It exits 0 whatever happens.
Its last log line, untested/1 {counts}, is what the weekly list is built from: hunks, killed, survived, survived that only remove, unchecked, over budget, flaky, error, file and mutant fallbacks, covered, trailers and their problems, and seconds.

Measured on this desk

Change	Hunks	Outcome	Time
#607 (`9ca55015`), kernel	5 (4 in the mutant lists, 1 unchanged to Python)	5 killed	14 s
#606 (`fb601dba`), `tools/check_workflows.py`	12 (6 in the mutant lists)	4 killed, 8 UNCHECKED over budget; 5 went to their file's revert	1,262 s, the budget binding. The desk was running the mutant run beside it
#604 (`5c4602d3`), `tools/hooks/shell.py`	11	11 killed, 2 by their file's revert	75 s

Records

0166, "A changed line of Python no test notices is reported for a month before it can fail". The Factory Builder gave the number on 2026-11-08: #628 took 0164 and #632 took 0165. Twenty-three Boundaries rows, each naming its tests in tests/test_untested.py and its mutants.

Checks

Check	Result
`tests.test_untested`	25 OK, 7 s
The 50 new mutants	50 of 50 killed at `cfb6b30`; all 12 of CI's shards pass at `a65ddb0`
`make verify` on an export of `a65ddb0`: record 0166, main merged in at `f1e3c7b`	1705 tests OK (expected failures 8)
CI at `a65ddb0`	Every check passes but `untested`, its one-time run, as above

Reads

Reviewer	Head	Major	Minor	Info
claude-review	`347315f`	1	5	1
claude-security	`347315f`	0	3	1
agy (Argon)	`347315f`	1	0	0
Codex (bot)	`347315f` to `24d7e68`	0	14 (P2)	0

Muse was not run: it is out of quota until 2026-11-09T00:00Z, and its read is owed then. Codex reached its usage limit at cfb6b30 and gave no read of it.

Finding	From	Fixed in
A hunk that only removes is put back unchecked; tests/ left out, the parse and import guards and the 40-line limit held by no test	claude-review	`a13632b`, each with a test and a mutant
A trailer on a hunk never run is a false problem	claude-review, Codex	`a13632b`
A large hunk whose mutants all ran out of budget is not counted over budget	claude-review	`a13632b`, then `553edae` for some of them
The runner's file-command variables reach the head's tests unheld by a test	claude-security, claude-review	`a13632b`
Code inside the `MUTANTS` assignment counts as data	claude-security, Codex	`a13632b`, `359cb79`
A file revert's kill is counted as the hunk's own	claude-security	`a13632b`: `killed_by_file`
The head's tests can forge the log line the weekly list reads	claude-security (info)	The record says so; the switch must move the verdict out of the head's process tree
This PR's own `untested` run fails	agy	Kept, as decision 0023 gives it; said in the workflow and the record
A fallback tried but not giving the outcome is not counted	Codex	`359cb79`
A kill's confirmations, an import probe, or the suite's discovery could outlast the step	Codex	`a484f99`, `ad01122`, `6f36c4a`
A mutant of a deferred annotation; a CRLF file's edit is STALE; a hung mutant beside killed ones is killed	Codex	`ad01122`, `6f36c4a`
A condition regex that backtracks	SonarCloud	`a13632b`
A form feed or U+2028 in a string moves a revert onto the wrong line; a long test name hides a crash; a trailer's span is counted line by line	Codex	`793ba05`, `cfb6b30`, each with a test and a mutant
The new crash mutant repeated an older mutant's text, which went STALE in shard 1	CI	`a65ddb0`

🤖 Generated with Claude Code