cmux-next swift test: fail when fewer tests report than were discovered
cmux-next swift test could read as a pass over tests that never ran.
The wrapper failed a run only when Swift Testing's Test run with N tests ... summary line was missing entirely. But that line counts only the tests that got to report. When a run ends part-way and SwiftPM still prints a summary, the summary is smaller and the run looks green — so exit 0 plus a summary line was accepted.
What changes
A summary line no longer excuses a test that started and never reported. Every test that prints a started. line must print a result line; if any does not, the run fails even when swift test exited 0, and one ::error names them.
The expected number comes from the log itself. That is the whole point of this version: an earlier draft asked swift test list how many tests there should be, and review blocked it twice. The second block was correct in a way the first fix had not addressed — proving that a selection changed the count list returns does not prove list selected the same tests swift test ran. If list honoured --filter but ignored --skip, the count still shrinks, the proof still concludes "honoured", and a complete run gets failed. That would have been a base red on nearly every swift test job, produced by the lane whose job is clearing base reds. Cardinality is not set equality and no cheap proof makes it one, so the second tool is gone: no swift test list, no filter-semantics assumption, no added wall-clock.
It also fixes a latent bug in the existing unfinished-test report. A test that ends inside withKnownIssue prints ━ Test ... passed, a glyph the old result patterns did not match, so such a test was already being listed as never finished. A result line is now recognised by its words rather than its glyph (passed after, failed after, skipped, or a parameterized function's case-count form), which covers ✔, ✘, ➜ and ━ alike.
This hardens an existing job's verdict. It adds no CI gate, no job, no step and no new test file: scripts/cmux-next/tests/swift-test-summary.test.sh is already registered at cmux-next.yml:1592, and the new cases go in it. The script change is 30 lines.
Changelog
none
Red, then green
bash scripts/cmux-next/tests/swift-test-summary.test.sh — the existing suite plus five new cases, driven against a stub swift, so it needs no toolchain and runs in seconds.
With the base sampler script and this PR's test file:
a run whose summary hid two unfinished tests exited 0
On this head: ok swift-test-summary, every case passing, the pre-existing ones included.
The invariant is measured in both directions on real fleet logs, not assumed:
| measurement | |
|---|---|
| does it false-fail? | across the 2,055 complete suite runs of a full fleet run, every test that started also finished — no exception |
| does it catch the real truncation? | on the truncated run of #25530 it separates 3,443 started from 916 finished |
Four mutations, each caught: drop the exit wiring; never check when a summary is present; tolerate one unfinished test; stop treating a known-issue finish as a result.
Observed / Not run
Observed: the new cases red on base and green on this head; the four mutations each turning the suite red; the two fleet-log measurements above; prepush-gate.py reporting no new violations; bash -n on both files.
Not run: the guard has not yet run inside a real cmux-next swift test job — that job does not route to this PR, which changes only shell scripts. This version is why that is now an acceptable risk rather than a gamble: it reads only the log the job already produces, so there is no second tool whose behaviour on the real invocations could differ from the fixture's. The remaining exposure is the result-line vocabulary, and that is measured against 2,055 real suite runs above.
Related: #25530 fixes one truncation cause (an unowned runtime read after deinit, which aborted the helper on signal 6). It is not the only one — a run on #25530's own head still ends through exit() from swift_task_asyncMainDrainQueue with 2,722 tests started and never finished, which is the symptom originally reported. That second cause is being tracked separately; this PR is the half that makes any such run fail instead of reading as a pass.
🤖 Generated with Claude Code
Agent session: agent session