PR #29650 - manaflow-ai/cmux

Leo, 2026-10-31: "FOR OUR NIGHTLIES ARE THEY END TO END ON OUR FLEET? WE CAN'T BE RELYING ON BLACKSMITH THAT CUTS OUT" and "i mean the linux things cant our minis just run them".

What was wrong

Nightly run 41234993713 (tip 3017ef19) queued decide on blacksmith-32vcpu-ubuntu-2404. Its runs-on had only that label and vars.LINUX_RUNNER as a size switch, so there was no fleet route. Every other nightly job waits on decide. #29130 and #29436 moved the other light nightly jobs onto portable placement, but decide can't use that: it is the run's first job, so no picker or placement job can run before it.

test_light_nightly_jobs_stay_on_linux_while_the_overflow_switch_is_on only checked that these jobs stay on Linux, so the literal pin passed.

Change

decide runs-on: attempt 1 of a nightly-next push (CI_PR_POOL_OWNED == '1') now takes vars.CI_NIGHTLY_NEXT_BUILD_RUNNER. That is glaeda's nightly slot (glaeda-nightly-std-xcode-26.6), a runner on each mini that takes only nightly jobs, ahead of PR jobs.
- Every other case still gets the Blacksmith Linux set as before: attempt 2+, a dispatch, main, forks, or the variable unset.
- decide is a checkout, github-script and two stdlib Python helpers (nightly-publication-marker.py, nightly_build_inputs.py, both 3.9-clean). It holds no secret and does nothing Linux-specific.
Fallback: the main-branch owned-pool rescue already watches attempt 1 of every nightly-next push (nightly_target()), and its job_pool() matches the nightly slot through LANE_LABEL. When no slot takes decide within CI_OWNED_POOL_RESCUE_SECONDS (30 s here) and nothing in the run has started, it cancels and re-runs. Attempt 2 then takes Blacksmith Linux. That is the stall-aware failover. A started nightly is never cancelled (assess(reserved=True)).
.github/runners.json: adds nightly.yml#decide as portable-checks, with a picker_followup saying why it reads the slot variable rather than a picker output. runner_routing_table.portable_order() accepts the nightly slot as the mini term for nightly jobs.
Ratchet (the test that should have caught this): test_no_new_nightly_job_rides_blacksmith_linux_alone fails any nightly.yml job whose runs-on reaches Blacksmith Linux with no fleet term (slot, side lane, or a picker/placement owned output), unless the job is in BLACKSMITH_LINUX_ONLY_DEBT. That list must shrink: it also fails on an entry that has gained a fleet route. On base it fails with ['decide'].
test_decide_takes_the_owned_nightly_slot_then_blacksmith_linux (in test_nightly_trusted_routing.py) evaluates the expression for each case.
Rescue (owned_pool_rescue.py): nightly_watched() now also judges decide when it waits on a glaeda-nightly-* slot. The copy that runs is main's (ci-owned-pool-rescue.yml checks out ref: main), which already judges every job_pool() job. This keeps feat-cmux-next's copy in line, with a test.

Review: router-review gpt-6.1-sol xhigh BLOCK at 5ef6c4af (the rescue predicate), fixed in 9aa189ad, then SHIP at 9aa189ad253b0cdd9e12ce4f31fcdbf44bc2049c. Follow-up: watch side/dev labels too if CI_NIGHTLY_NEXT_BUILD_RUNNER ever names one; today it names the nightly slot.

Not in this PR

The other 33 Blacksmith-Linux-only nightly jobs are in the debt list. Moving them needs a placement step ahead of each; the ones after build-nightly-app can reuse #29436's light placement. They are a follow-up, and the ratchet keeps the list from growing.

Checks run

These test files pass directly: test_nightly_trusted_routing, test_ci_nightly_off_blacksmith, test_runner_routing_table, test_ci_macos_xcode_selection, test_ci_owned_pool_sources, test_ci_nightly_no_hosted_mac, test_ci_linux_single_size. runner_routing_table.py exits 0. test_ci_blacksmith_linux_failover fails the same way on base.

Performance

metric	before	after	delta	harness
`decide` Blacksmith minutes per nightly-next push	~1 (32 vCPU, about 10 s busy plus setup)	0 when a nightly slot is idle	about -1 per push	run 41234993713 timings

Agent session: agent session

🤖 Generated with Claude Code