#28594 ci(cmux-next swift test): a stall is no finish AND idle processes; a cut-short run counts what it finished

The cmux-next swift test broad pass is cut short on PRs: the stall watchdog ends it with exit 143, ~225 tests open and no summary. A cut-short run is never a base red, so every Swift PR's test red is unreadable. This PR fixes the watchdog's half of that. The sharding of the pass is #26809, and the runner starvation is the glaeda holder fix (teamleaderleo/glaeda#1763, on stable since 23:11Z on 2026-10-30).

Why the runs were cut: starvation, not a hung test

I read the logs of #28497 run 41123927916 (attempt 1, job 124298297201; attempt 2, job 124338094462) and of #26919's run at 1d53e262 (job 124328762013). All three ran on blacksmith-6vcpu-macos-26.

• Tests were slow, not stuck.
  • Tests that take under a second on an idle mini finished after 78-317 s (passed after 78.920 seconds, 269.6 s).
  • The throughput fell from ~3,000 finishes a minute to ~130.
  • The last finishes came in bursts 20-80 s apart until the 313-343 s gap the 300 s rule fired on.
• No common hung test.
  • The open lists hold 223-228 tests: those started in the last burst, plus parameterized tests from the start of the run.
  • The newest open tests differ from run to run.
• The main thread was not hung either. The watchdog's sample of the test helper (5.1 GB footprint) shows:
  • attempt 2: in malloc while it creates a task in NotificationCenterService.start (a WorkspaceChecklistTests setup);
  • attempt 1: in SidebarListView.setWindowVisible;
  • #26919: idle in mach_msg.
• The same pass finishes on a mini. The tip Swift suite runs on cmux13s/cmux9s minis finished ~8,000-9,000 tests and then hit the 1,500 s whole-run cap while still finishing tests.

What changes (scripts/cmux-next/swift-test-with-hang-sampler.sh)

• A stall needs idle processes.
  • "Stuck" means no test finished for CMUX_NEXT_TEST_STUCK_SECONDS (300) and the run's test processes used less than CMUX_NEXT_TEST_IDLE_CPU_SHARE (0.05) of one core in that time: a deadlock, or a wait that never ends.
  • A run that is busy but silent gets CMUX_NEXT_TEST_BUSY_STUCK_SECONDS first (default 3x the stall limit, so 900 s). The log says once that the run is slow, not hung.
  • CPU is read with ps -o time= over swift test and every process under it, summed per process.
  • With no reading, the rule falls back to the old finishes-only rule.
  • CMUX_NEXT_TEST_HANG_SECONDS stays the hard ceiling on the whole run.
• A cut-short run says what it got done.
  • When the stall rule, the whole-run limit, a crash, or a missing summary ends the run, the wrapper prints one ::error title=cmux-next swift test cut short:: line and a step summary heading.
  • The line counts the tests that finished (passed, failed and skipped), names the failed ones, and names the ones that never finished.
  • Every finished test's own ✔/✘ result line stays in the log. scripts/ci/test_reds.py reads those lines and compares each failing test by name with the base.
  • Its existing cut-short rule (never finished, swift test stuck, no "Test run with" summary line) still marks the run as cut short. So base_red_excuse.py still refuses to call it a base red, as before.

Tests (scripts/cmux-next/tests/swift-test-summary.test.sh, run by the cmux-next checks job)

• A busy but slow run is not cut. A stub swift spins one core for 6 s with no finish, under a stall limit of 2 s, then finishes whole. It is red on the old wrapper ("the watchdog cut a slow run whose test process kept using CPU") and green here.
• A busy spin is cut at the busy limit. It names its limit, says the processes kept using CPU, and names the open test.
• An idle hang is still cut at the stall limit. The error now says the processes sat idle.
• The cut-short counts and names, for a stall and for a run with no summary. The no-summary case is red on the old wrapper too.
• The whole file passes locally in 47 s, with no Swift.

Performance

Harness: the cmux-next swift test job. The "before" figures are numbers that already exist. The "after" column is the swift-test wrapper's own test (swift-test-summary.test.sh), run on a fleet mini (cmux14, macOS) and on Linux. This PR's own cmux-next swift test job had been queued since 23:58Z behind 60+ queued swift-test jobs. The job-level after numbers come from the first base and PR runs on the merged wrapper.

metric	before	after	delta	harness or run link
broad pass ended by the 300 s stall rule while tests still used CPU	3 of 3 runs read (#28497 attempts 1 and 2, #26919)	0 in the harness: a run that keeps one core busy with no finish for 3x the stall limit is not cut and then finishes whole (macOS mini and Linux); an idle hang is still cut at the stall limit	-3 of 3 cut for busy-silent spells under 900 s	before: runs 41123927916 (jobs 124298297201, 124338094462) and 41118922966 (job 124328762013); after: swift-test-summary.test.sh on cmux14
swift-test job p50 (24 h to 10-30 ~21Z)	1,800-2,100 s (600-800 s before ~10-29 09Z)	unchanged by this PR (it starts no test and builds nothing); the starvation fix is glaeda#1763	n/a here	fleet job telemetry, 443 runs
runs reaching the 1,500 s limit	306 of 443 (69%)	unchanged by this PR (the whole-run cap stays 1,500 s); the cap is addressed by sharding in #26809	n/a here	same
failing tests comparable by name in a cut-short run	the finished tests' result lines, plus an incomplete line naming the open tests	the same, plus one cut short line with the finished, passed, failed and skipped counts and the failed and open tests by name	+1 summary line	swift-test-summary.test.sh

Incremental builds and the SSD cache: the change adds no build. The wrapper runs on the package and tests that the job's swift build --build-tests already built, whether incremental on an owned mini or warm from the shared compile cache. ps is read once a poll (10 s).

New timer, and how it stops: the CPU reading runs inside the watcher's existing 10 s poll loop. It ends with that loop: the watcher is killed when swift test exits, or exits after it samples and kills a stalled run.

Risk: CI test wrapper only. A real hang whose process keeps spinning now takes 900 s instead of 300 s to be cut. An idle hang is still cut at 300 s.

Agent session: agent session

🤖 Generated with Claude Code