cmux-next swift test: a truncated run can no longer read as a pass

Closed
TE
10d ago
+1105-72

What

Both halves of the ask: the cmux-next swift test job can no longer read as a pass when it ended
early, and it no longer ends early.

The process does not crash. swift-testing's async main runs CFRunLoopRun on the main loop, AppKit
calls CFRunLoopStop(CFRunLoopGetMain()) to wake an event wait, the stop lands on that outermost
activation, swift_task_asyncMainDrainQueue returns, and main falls through to exit(0). Every
test that had reported is already in the log, so SwiftPM still prints a Test run with N tests
summary — the summary counts only what reported. With swift test exiting 0 and every reported
test green, the job read as a pass with thousands of tests missing. HomeLaunchRevealTests
never reported for four days and nobody saw it (#23680, #23704, runs back to 2026-10-22: 2192
counted against 4736 started).

The cause

HomeOpenFocusesComposerTests is @Suite(.serialized) with exactly two tests. From the swift test
job 121471396706 of this PR:

09:02:16.2173 ✔ Test aClickLeavesTheRunLoopItRanInRunning() passed after 223.890 seconds.
09:02:16.2174 ◇ Test openingAConversationFromTheListPutsTheKeyboardInItsMessageBox() started.
              ... 28 lines ...
09:02:16.2193 cmux-next test process: exit() called

The first test wraps its one click in inANestedPass and passes. The second calls click seven
times bare, starts, and the process ends 2303 tests short. The atexit tracer can never name the
caller — by the time it runs, the stopper's frames are gone — which is why three days of stack
reading got nowhere.

Nothing else in flight at that instant can post an NSEvent: the other tests that had just started
are a socket-router call (listWithNoAcpmuxIsEmpty), two DaemonStore data cases
(oneStatusChangeVisitsOneWorkspace) and colour maths (aWindowsAccentTakesTheHueOfTheWorkItShows).
Every raw NSApp.postEvent in Sources was traced too and none is reachable from any test:
DebugMouse.swift:135 and DebugKey+Sequence.swift:47 are #if DEBUG and reached only from
AppControl's control socket, which no test builds; WindowHintController.swift:155 is behind a
live hints.show session, which no test starts; AddressBarView+Debug.swift:112 is behind
debug.omnibar, likewise. There is no bare CFRunLoopRun in the package, no NSApp.stop anywhere,
and the four CFRunLoopStop(CFRunLoopGetMain()) sites in CmuxNextControlTests are in a target
this job skips.

The fix

Isolating only the postEvent was never enough. click also runs document.mouseDown, which
drives the list's own press-and-drag tracking, and then an NSApp.nextEvent drain loop in
.default mode. "postEvent, then nextEvent, in any run loop mode" is verbatim the shape this
repository already blames for ending the process (KeyDownDispatchTests.swift:18,
ActionRunTimingTests.swift:324, which both avoid it by poking currentEvent).

So the nested pass moves into click and key instead of staying each caller's job, which
removes the whole class of "someone called click bare". The two passes take different run loop
modes, because aClickLeavesTheRunLoopItRanInRunning nests one around a click that now nests its
own, and a block left in one mode must never be run by the other's activation. That test keeps its
assertion and finally gains its meaning: the pass ends .finished because the click absorbed its
own stop, not because this caller remembered to wrap it.

The gate

report() in scripts/cmux-next/swift-test-with-hang-sampler.sh already caught the no-summary
shape. It now also fails when a summary is present but any test the log shows starting never
finished, or when the summaries total fewer tests than started.

Three traps, each one a false failure I hit and fixed:

  • Not equality. A skipped test prints no started. line, so a legitimate summary runs ahead
    of the starts.
  • Summary counts are summed. One log can hold a summary per test target — the isolated-suite
    step holds dozens — while starts are counted across all of them, so comparing the last summary
    alone fails a log holding two whole runs.
  • Finish markers are read anchored, and there are four of them. A marker matched anywhere lets
    a test's own output (debug: ✔ Test hung() passed…) self-report as finished. And a finish is not
    only ✔/✘: a known-issue finish carries ━. The live daemon-tests job of this PR read
    ━ Test aRunCutOffMidwayFinishesWithoutDuplicates() passed after 8.359 seconds with 2 known issues.
    as unfinished and failed a run of 386 tests whose own summary counted all 386. Separate anchored
    patterns, never one bracket set: awk reads [✔✘➜━] as the set of bytes those characters are
    made of, and each ends in a byte another also contains, so match() lands mid-character and every
    line matches. That bug made the gate report nothing at all, and the existing red fixture caught it.

scripts/cmux-next/tests/swift-test-summary.test.sh holds the regression cases, and the checks job
already runs it.

Deliberately not added

A guardrail refusing new raw NSApp.postEvent / NSApp.nextEvent / CFRunLoopStop lines. CLAUDE.md
says not to add CI gates, and the gate above already catches a truncation and names the test that
never finished — the attribution this took three days to get by hand.

Observed / Not run

Ran here, on Linux:

bash scripts/cmux-next/tests/swift-test-summary.test.sh   ->  ok swift-test-summary
python3 -I scripts/cmux-next/prepush-gate.py              ->  no new violations
bash -n scripts/cmux-next/swift-test-with-hang-sampler.sh ->  ok

Both real job logs of this PR's earlier run were replayed through the extracted report():

  • daemon tests (121471396631): the "never finished" false failure is gone; the one genuine red,
    aUserRenamedHomeMarkerIsNotUnnamed() WorkspaceNameSourceTests.swift:18, still reports.
  • swift test (121471396706): the truncation still fires, through the new branch —
    "the summaries count 420 tests in total but 6534 started and 2304 of those never finished".

Not run here: the Swift suite. Agents never compile or test on a Mac locally. This PR's own
cmux-next swift test run at this head is the evidence for the cause fix: a run that reports
every test it started is the thing being claimed.

Base findings at the earlier head, each also failing on #25394, #25395 and #25396 at the same time:
cmux-next generated files and cmux-next swift test (the truncation itself, which is what this
fixes). cmux-next daemon tests was the gate's own false positive, fixed here.

Changelog

none

🤖 Generated with Claude Code