#16728 ci: cmux-next Mac jobs take a mini on attempt 1 only when one is idle now

Summary

Most cmux-next.yml runs wait out a dead first attempt. Of 60 runs on 2026-10-02 (UTC), 49 (82%) needed attempt 2. In every attempt 1 sampled (10 of 10; for example runs 36972364136, 36972067034 and 36971945422), the three Mac jobs (swift test, Release compile, scheme compile) never got a runner. They sat queued for 99 to 151 s until scripts/ci/owned_pool_rescue.py's 90 s budget ran out, and the rescue cancelled the run and re-ran it.

The cause is that side-lane workflows have no picker. Their Mac jobs take vars.CI_SIDE_LANE_RUNNER (the std minis' glaeda-side-* label) on attempt 1 of any trusted run, without looking at the fleet. When the side runners are busy, attempt 1 is pure queue time.

With this change, a new macos-placement job in cmux-next.yml runs before the Mac jobs, on Linux, and reads the runners live through the route App, the same way ci-macos.yml's admission-placement does. It puts each Mac job on the side label only while an owned side runner is idle now, one job per idle runner, in priority order: scheme compile first (it keeps DerivedData on a mini and builds cold on Blacksmith), then Release compile (only a mini compiles the CEF shim), then swift test. Any job no idle runner takes uses the fallback it already had (vars.MACOS_RUNNER_26, else blacksmith-6vcpu-macos-26). That's the label attempt 2 takes today, without the 90 s wait and the cancel.

Minis first is preserved. The owned runner stays the first choice, and a job falls back only when no side runner is idle at that moment. The idle check is the one the main PR picker already uses for side lanes. I moved it out of light_side_lanes() into pr_runner_pool.idle_placement(), and both callers now use that one function. Nothing is copied.

Uncertain cases keep today's route. If the runners can't be read, the token can't be minted, or the placement job fails or is skipped, fallback_jobs stays empty. The Mac jobs then take the side label as they do now, and the rescue watches them. The Mac jobs carry !cancelled(), so a failed placement job never skips them.

The rescue still watches owned jobs. The owned-pool-watch marker moves from checks into macos-placement. It's uploaded unless every Mac job took its fallback. The sweeper decides which jobs to watch from each job's labels, so a mixed run is watched only for its owned jobs. A runner taken by another run between the read and the queue still leaves a job queued on the owned label, and the rescue moves it as before. owned_pool_rescue.py only gets docstring and comment updates; its behavior and SIDE_WORKFLOW_PATHS are unchanged.

Attempt 2 and later are unchanged. The runs-on expressions still gate the owned branch on github.run_attempt == 1, and the script itself decides nothing after attempt 1.

The placement job starts only where attempt 1 may take a mini: same-repository, manaflow-ai, owned pools on, attempt 1. A fork, a re-run or owned pools off starts no extra runner. A re-run needs no marker of its own, because its Mac jobs take the fallback and the rescue's side-lane watch ends at attempt 2. Where it does run, the cost is one short Linux job (sparse checkout, token mint, about three API calls) ahead of the Mac jobs. Other side-lane workflows can reuse scripts/ci/side_lane_placement.py (it takes SIDE_LABEL and JOBS), but only cmux-next.yml is wired here.

Testing

• Red first. tests/test_ci_side_lane_placement.py landed alone in 3e44fdafeca, and python3 tests/test_ci_side_lane_placement.py failed there with FAILED (failures=10, errors=2): the script and the wiring didn't exist yet.
• Green at 30758a877fb with the same command: Ran 10 tests ... OK. At 0683b131c89 there are 11 tests; the new one checks that the placement job is skipped for forks, other owners, owned pools off and attempts 2 and 3, that every run that skips it routes each Mac job off the minis, and that the Mac jobs still run and keep the side label when it is skipped. The tests cover:
  • a busy pool on attempt 1: every job falls back and none is queued for the rescue to cancel;
  • one idle runner takes the first job and four idle runners take all three;
  • only runners carrying the side label count;
  • attempts 2 and 3 decide nothing;
  • unreadable runners or a label that isn't a side label keep today's route;
  • decide() goes through pr_runner_pool.idle_placement();
  • the GITHUB_OUTPUT contract.
• Wiring: the tests evaluate each Mac job's runs-on in cmux-next.yml. A placed job or a missing placement gives the side label. A job in fallback_jobs gives the fallback. Attempts 2 and 3 and fork PRs always give the fallback. Each job's CMUX_NEXT_RUNNER equals its runs-on, and the marker step follows the placement.
• tests/test_ci_pr_runner_pool.py passes (225 tests; light_side_lanes() now goes through the shared function). So do test_ci_owned_pool_rescue.py, test_ci_fork_runner_routing.py, test_runner_label_policy.py, test_ci_workflow_guards_are_wired.py (the new test is wired into ci-guards.yml and registered in tests/test-execution.toml) and test_ci_guard_tests_emit_no_workflow_commands.py. actionlint is clean on cmux-next.yml and ci-guards.yml.
• scripts/ci/guards-local.sh: 77 steps passed and 4 failed. None of the 4 come from this change:
  • test_seed_derived_data.py's test_main_full_suite_admission_compiles_on_the_seed_pool_and_xcode fails the same way on a clean origin/feat-cmux-next checkout;
  • the fleet compiler-cache check fails on compile-app-host-test-product.sh, which this PR leaves unchanged;
  • shellcheck isn't installed on this Mac;
  • the guard's bundled Linux actionlint binary won't run on macOS (the local actionlint run above is clean).
• Not verified: ci.yml skips the ci-guards lane for PRs into feat-cmux-next, so CI won't run these tests on this PR. The red/green evidence above is from local runs. The new placement job's first live run also checks out the base branch's scripts/ci, so on this PR it fails open (the script isn't on the base yet) and the jobs keep the side label. The new behavior first shows up live after merge.

Changelog

none

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Stops cmux-next.yml's Mac jobs from queueing on a busy mini fleet on attempt 1, so trusted runs no longer wait out the owned-pool rescue's 90 s budget, get cancelled, and re-run (49 of 60 runs on 2026-10-02 needed attempt 2).

A new macos-placement job runs before the Mac jobs, reads the runners live through the route App, and puts each job on the minis' side label only while an owned runner is idle now, one job per idle runner, in priority order: scheme compile, release compile, swift test. Jobs with no idle runner take their existing fallback (vars.MACOS_RUNNER_26, else blacksmith-6vcpu-macos-26), which is the label attempt 2 takes today, without the wait.

• The idle check is pr_runner_pool.idle_placement(), factored out of light_side_lanes(); both callers now use the same function.
• The placement job runs only where attempt 1 may take a mini (same-repo, manaflow-ai, owned pools on, attempt 1); a fork, another owner, owned pools off or a re-run starts no Linux runner, and the Mac jobs there take their fallback and still run (!cancelled()).
• Everything uncertain (placement failed, runners unreadable, token unmintable) keeps today's route: fallback_jobs stays empty and the Mac jobs take the side label as before.
• The owned-pool watch marker moves from checks into macos-placement and uploads unless every job took its fallback; a runner taken between the read and queueing is still rescued as before, and attempt 2+ is unchanged.
• The placement job checks out the base branch's scripts/ci, so it fails open on this PR; the new behavior first shows up on the first run after merge.