PR #28605 - manaflow-ai/cmux

Summary

swift-test, daemon-test and generated-files now follow the routing order on attempt 1 and on the rescue's re-run. They take an owned GUI or session Mac first. They go to Blacksmith macOS last, at the largest size that can start them now.

Run 41123927916 (#28497) ran swift-test on blacksmith-6vcpu-macos-26 on both attempts. Tests that take under a second on an idle machine took 78-227 s there, and the 300 s stall watchdog cut them. Attempt 1's placement said: "glaeda-gui-std-xcode-26.6 has no room (54 queued, 11 online, none idle), so the GUI jobs take blacksmith-6vcpu-macos-26". Each attempt reached 6 vCPU through a fixed leg, not through a size anyone weighed:

Attempt 1: needs.macos-placement.outputs.runner != '' && 'blacksmith-6vcpu-macos-26'. Placement always writes runner, so every declined GUI job took this leg before gui_overflow was read.
The rescue's attempt 2: rerun_main() wrote no gui_runner. Its gui_overflow was always 6 vCPU, so an idle owned GUI Mac was never weighed.

Changes:

next_macos_cascade.gui_overflow_label() names 12 vCPU only when it can start the GUI jobs now. That means it is not stalled or dead, has no freshness work queued, has a free slot for each GUI job, and the shared 60 vCPU pool fits them. Otherwise it names 6 vCPU. It also names 6 vCPU when the counts or the snapshot can't be read.
side_lane_placement main() and rerun_main() write gui_overflow from it, after counting the slots the run's own cascade just took.
rerun_main() also writes gui_runner, session_runner and the free labels, naming only a runner that is idle now. The rescue still watches attempt 2, so a free runner lost to a race gets moved once more.
cmux-next.yml: the runs-on and the CMUX_NEXT_RUNNER copy of all three GUI jobs drop the runner != '' leg, the run_attempt != 1 && 'blacksmith-6vcpu-macos-26' leg and vars.MACOS_RUNNER_26. They now read the placement's role route, then gui_overflow, from the current attempt only. The attempt-1-gated owned role label stays. They end on 'blacksmith-6vcpu-macos-26' as the last resort, which covers a person's re-run, a carried placement and a fork.

Honest limit: that day 12 vCPU was stalled ("blacksmith-12vcpu-macos-26 0 in flight + 158 queued of 5 (stalled: oldest queued 1431 min)"). With this change, run 41123927916's overflow would still have been 6 vCPU. What changes is that the bot's re-run takes an idle owned GUI Mac first, and that 12 vCPU is used whenever it is live. 6 vCPU starving swift-test is not fixed here. That needs fewer test workers on small runners, or a weighting that accounts for 6 vCPU running swift-test slower.

Not touched: the swift-test steps and scripts/cmux-next/swift-test-with-hang-sampler.sh (#28594), and the swift-test step sharding (#26809). #28249 and #28391 also edit rerun_main()'s output lines. Whichever lands second takes both sides.

Testing

Every new test fails on this base (cd2b03c1893) and passes on the head. The tests are in their own first commit.

python3 -I tests/test_ci_runs_on_fallback.py: new test_no_fixed_blacksmith_leg_comes_before_the_gui_overflow (base: "cmux-next.yml swift-test runs-on: needs.macos-placement.outputs.runner != '' && 'blacksmith-6vcpu-macos-26'") and test_the_overflow_rule_catches_what_it_is_for.
python3 -I tests/test_ci_next_macos_cascade.py: new GuiOverflow (base: no gui_overflow_label). 12 vCPU when it moves and fits. 6 vCPU when it is stalled (run 41123927916's figures), dead, full, short of a slot per job, held by a pool queue or reserved, or when the counts or snapshot are unread.
python3 -I tests/test_ci_side_lane_placement.py: new test_the_rescues_full_rerun_takes_an_idle_gui_runner_first (base: no gui_runner written) and test_the_rescues_rerun_takes_the_owned_gui_runner_or_the_overflow_its_placement_found (base: an idle GUI runner still got blacksmith-6vcpu-macos-26).
Updated: the pinned generated-files expression in tests/test_cmux_next_checks_workflow.py, plus the expectations that change from 6 to 12 vCPU while 12 vCPU is free and moving.

Observed / Not run

Observed: all of test_ci_side_lane_placement (75), test_ci_next_macos_cascade (97), test_ci_runs_on_fallback (19), test_cmux_next_checks_workflow (165), test_seed_derived_data, test_cmux_next_visual_gate and test_ci_session_label_routing pass. test_ci_change_areas fails one test, test_repository_helpers_route_by_where_they_run, and it fails the same way on base. actionlint has nothing on the changed lines. The pre-push gate is clean.

Not run: a live CI run with a declined GUI queue. The first PR run after this lands shows its placement line in the step summary.

Changelog

none

Agent session: agent session

🤖 Generated with Claude Code

Summary by cubic

swift-test, daemon-test and generated-files now follow the macOS routing order on attempt 1 and on the bot's re-run: an owned GUI or session Mac first, Blacksmith macOS last at the largest size that can start the jobs now.

The GUI overflow is now a decision, not a fixed 6 vCPU move: gui_overflow_label() names 12 vCPU only when it can start every GUI job now (not stalled or dead, a slot per job, shared pool fits), else 6 vCPU.
rerun_main() writes gui_runner, session_runner and the free labels, so a re-run takes an idle owned Mac before any Blacksmith runner; with the merged dead-side-queue routing it still writes runner for freshly allocated owned jobs and deferred_runner.
The runner != '' and run_attempt != 1 legs that sent declined GUI jobs straight to 6 vCPU are gone; 6 vCPU is only the last resort. A placement naming neither a GUI route nor an overflow gets the last resort from the place step (fork-aware via GUI_LAST_RESORT: macos-26 for forks, blacksmith-6vcpu-macos-26 otherwise), so its GUI jobs never join the unmeasured owned queue.

Honest limit: on run 41123927916 12 vCPU was stalled, so that overflow is still 6 vCPU; what changes is that re-runs take an idle owned Mac first and 12 vCPU whenever it is live. 6 vCPU starving swift-test is not fixed here.