PR #29490 - manaflow-ai/cmux

Carries #26809 by its owner's commits. Leo: "Speed up these iterations."

This is #26809's branch at b777afc8 with origin/feat-cmux-next merged in (a clean merge, no hand-resolved conflict). The PR's own diff is identical to #26809's: every +/- line matches, only hunk context moved. #26809 sat unlanded since 05:03Z because its cmux-next swift test job waited on the GUI pool; that job has since run in full at b777afc8 (job 124552356302): 15 shards, none cut short.

Performance (fleet impact)

metric	before	after	delta	harness or run link
PR `cmux-next swift test` success rate	about 1/290	measured after merge	pending	ci-dash jobs.csv, 24 h to 13:50Z 2026-10-31
Mac minutes per day on `cmux-next swift test`	about 11,000	measured after merge	pending	same

#26809's description

The "cmux-next swift test" broad pass on feat-cmux-next is cut short. One shared test process runs every suite, so a few main-thread-heavy suites (frame benchmarks, real WebKit downloads) and the windows earlier suites leave open starve every main-actor test after them. The pass then hits its time limit with hundreds of tests started and never finished. Run 40705037968 is an example: 466 unfinished, throughput fell from about 2,000 tests a minute to about 100, and 428 windows piled up in one process. The other cut-short cause, a CFRunLoopStop that reaches swift-testing's outermost run loop and ends the process with exit(0), is fixed by #25861's run-loop guard and is not in this PR.

1. Main-thread-heavy suites run in their own processes

scripts/cmux-next/main-thread-heavy-suites.py derives the list from the test sources on every run. It picks every suite whose file commits frames itself (CATransaction.flush(), displayIfNeeded(), a bench's drawFrame() or uses a bench helper type declared in such a file, plus RemoteVideoPageTests. A new frame-driving suite is covered with no list to edit.
The broad pass and the changed-suites pass both skip them.
A step of their own runs each one in a fresh process (hang sampler, 45 min). A hang or a red there is named in that step, and the rest still run. The step covers only the selected targets' heavy suites.
The job timeout goes from 90 to 120 minutes to fit that step. The per-test stuck watchdog is unchanged.

2. The broad pass runs as several test processes

scripts/cmux-next/swift-test-shards.py splits the selected targets by target, and a large target by suite-name prefix.
- Each prefix level ends in a negated class, so every suite lands in exactly one shard. A test checks this against every real suite and against made-up names.
- CmuxNextAppTests.Cloud* gets a shard of its own: its brokers and daemons run under 30-60 s deadlines.
It packs the pieces into about twice as many shards as run at once and starts the largest first. Each shard runs through the hang sampler with swift test --skip-build --ignore-lock.
The number run at once comes from the runner: half its cores, and no more than its memory holds at 4 GB a process. CMUX_NEXT_SWIFT_TEST_WIDTH overrides it.
Shards start one at a time. The next one starts once the last shard has printed its first test line, has exited, or 180 s have passed. A shard that failed on SwiftPM's "database is locked" before running any test is started again, once.
Each shard's log prints as one group, a red shard is named, and the step summary gets a table of shard, tests, seconds and exit.

3. The test process catches SIGPIPE, as the app does

A shard died on signal 13 with 19 tests never finished: some test wrote to a pipe whose reader was gone. The app catches SIGPIPE at launch (ChildSignalDefaults.installAppSignalPolicy()), so there that write fails with EPIPE. The test process never ran that step.

The test bundle now installs the same caught handler when it loads (Tests/Support/CmuxNextTestExitTrace/CmuxNextTestSignalPolicy.c). Exec resets the handler in children.
The first three catches print a one-line note on stderr (async-signal-safe write(2) only; a stack print can deadlock in a signal handler), so the note lands next to the writing test's own "started" line.

Performance

Harness: the cmux-next swift test job's broad pass on a fleet mini, same job and same pass on both sides.

metric	before	after	delta	harness or run link
broad pass wall time	over 1,500 s, cut short at its limit	571 s, every shard with a summary	about -62% or more	before: run 40705037968, job 122875223361; after: run 40723548016 (15-core mini, 14 shards, 7 at once)
tests finished per minute	about 300 average (about 2,000 early, about 100 late)	about 1,000	about 3.3x	same
tests started and never finished	466	0 outside shards with a named crash	-466	same

Tests

tests/test_ci_swift_test_main_thread_isolation.py (ci-guards) keeps the heavy split and tests the detector. It fails on today's workflow.
tests/test_ci_swift_test_shards.py (ci-guards) covers:
- every suite lands in one shard;
- Cloud gets its own shard;
- shard width;
- a fake swift test that holds a lock while it starts: the gate and the retry each fail on the code before them.
TestProcessSignalTests.aWriteToAClosedPipeFailsWithEPIPEAndTheRunGoesOn: on today's code the write ends the test process.

Risk: test harness and CI only, no product code.

Shard width, heavy-step limits (3727a9ce)

SidebarIncrementalMatchesRebuildTests, SidebarIncrementalMatchesRebuildGapsTests and ClassicUpgradeHugeSimulationTests run in the heavy step (ALWAYS in main-thread-heavy-suites.py). In shared shards they starved the main thread until the stuck watchdog ended shards 7 and 11 (run 40874507839).
Heavy-step watchdog limits per suite come from main-thread-heavy-suites.py --runs: 600 s run / 300 s with no test finished, or above a suite's own .timeLimit. ClassicUpgradeHugeSimulationTests (15 min limit; 412 s alone, one test open 572 s in a shard) gets 1200 s / 1020 s. The heavy step's timeout is 60 min and the job's is 135 min.
Broad-pass shards at once: a quarter of the cores (3 on a 14-core mini, was 7), still 15 shards.

metric	before (run 40874507839, 825cea78)	after (run 40896449485, e3c31bf6)
shards at once	7	3
shards cut short (exit 143)	3 (4, 7, 11)	0
broad pass wall time	13.0 min	8.1 min
heavy step wall time	9.6 min	14.7 min (3 more suites)
swift test job wall time	35.0 min	31.1 min

Agent session: agent session

🤖 Generated with Claude Code