#26017 ci(nightly-next): the app build takes an idle owned Mac, else a moving Blacksmith label

Row 3 of the 2026-10-27 nightly path audit: on nightly-next, build-nightly-app read vars.CI_NIGHTLY_NEXT_BUILD_RUNNER alone for every attempt. That was one label with no fallback, and a rescue re-run went back to the label that had refused it. Runs 40446901975 and 40448164753 died at "Set up runner" on glaeda-dev-std-xcode-26.6. Nothing re-ran them, because the owned-pool rescue doesn't watch glaeda-dev-*. Run 40453543604's attempt 2 was then refused at "Set up runner" on the side label.

Leo's routing order: Linux, then owned Macs, then Blacksmith macOS last ("IT WAS ALWAYS FUCKING LAST RESORT"). No job waits on one reserved machine.

Change

• New Linux job pick-nightly-build-runner runs scripts/ci/nightly_build_runner.py right before the build queues. It outputs:
  • owned: the first label in CI_NIGHTLY_NEXT_BUILD_RUNNER, then CI_SIDE_LANE_RUNNER (or a wider class of it) that has an idle online runner now. This uses side_lane_placement.compatible_labels and pr_runner_pool.live_owned_free, the same rule side lanes use. Only labels the owned-pool rescue watches are offered (pr_runner_pool.persistent). So glaeda-dev-* is skipped and the side label gets its turn.
  • blacksmith: the Blacksmith macOS 26 label whose queue is moving (nightly_sign_runner.pick, the picker the signing jobs already use). Never empty.
• build-nightly-app on nightly-next takes owned on attempt 1, and blacksmith otherwise. That covers a busy fleet and every re-run, including the rescue's re-run after a "Set up runner" refusal or a stuck owned queue. A failed pick falls through to vars.MACOS_RUNNER_26 || 'blacksmith-6vcpu-macos-26'. No GitHub-hosted Mac and no macos-15 anywhere, past the fork branch.
• Owned is only offered where the rescue watches: attempt 1 of a nightly-next push with owned pools on.
• Main, RC, dispatch and fork routes are unchanged. prepare-nightly-variant-primary already moved to Linux (#25885), so it needs nothing here.
• The concurrency groups, the Ghostty helper job and the os names are untouched.

Before / after: route options for build-nightly-app on a nightly-next push

	attempt 1	re-run (rescue or person)	dead label
before	1 label: vars.CI_NIGHTLY_NEXT_BUILD_RUNNER (glaeda-side-std, earlier glaeda-dev-std)	same label	waits up to 24 h, or dies at Set up runner
after	idle owned label (std side, then any wider class), else the moving Blacksmith label (6 or 12 vCPU, past stalled ones)	moving Blacksmith label	an owned label is used only if idle now; a refusal is re-run on Blacksmith

Measured queue waits from the last 40 nightly-next runs (jobs that got a runner, minutes):

• build-nightly-app on glaeda-side-std: median 0.05, p90 7.1, max 53.8 (n=22). That queue was taken whatever its load, so the tail is a busy fleet. The new route takes it only when a runner is idle.
• build-nightly-app on blacksmith-6vcpu-macos-26: median 9.9, p90 40, max 74 (n=12).
• glaeda-dev-std: 3 of 3 jobs failed at Set up runner with no fallback.

Guard tests

• tests/test_nightly_build_runner.py (wired into ci-guards' release-notary group). Its checks:
  • build-nightly-app's runs-on never reads CI_NIGHTLY_NEXT_BUILD_RUNNER directly.
  • Every combination of attempt, pick output and owned switch resolves to the picked owned label (attempt 1 only) or a blacksmith-* label.
  • The picker never fails its job.
  • A label the rescue doesn't watch is never offered.
• Updated test_nightly_trusted_routing.py and test_ci_nightly_off_blacksmith.py, plus the step registry entries.

Tests: the 75 nightly, runner and rescue test files were run through cmux-ci (fast-checks run 40463845215). The remaining failures (test_ci_change_areas, test_ci_fork_runner_routing, test_ci_workflow_*, test_nightly_legacy_publish_linux, two in test_runner_label_policy, test_ci_owned_pool_sources) fail the same way on base.

Known red: adds a second instance of base red test_dispatchable_jobs_with_external_credentials_use_environments (routing App token, same as pick-nightly-sign-runner). The fix, an environment for routing-token jobs, is trust policy and is being raised separately.

Risk: medium. It changes where the NEXT app build runs. A pick failure falls back to the existing Blacksmith default.

Agent session: agent session

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}