ci(cmux-next): stalled Blacksmith macOS gets no job; hosted macos-26 takes the overflow

Merged
TE
17d ago
+338-69

Blacksmith's macOS runners stopped starting jobs on 2026-10-20 (status.blacksmith.sh, "Delayed job starts on macOS runners", from 10:18Z; our queue stalled from about 04:00Z). About 80 cmux-next jobs sat queued on blacksmith-12vcpu-macos-26 and blacksmith-6vcpu-macos-26. The macOS cascade (scripts/ci/next_macos_cascade.py) kept adding to that queue. With no runner online, it counted every slot as free.

Run 39409184395's placement at 10:38Z logged: "blacksmith-12vcpu-macos-26 0 in flight + 4 queued of 5 ... blacksmith-12vcpu-macos-26 takes fast-tier". The janitor snapshot it read showed 12 vCPU's oldest queued job at 41 min, with nothing running.

What changes

Stall detection. This adds no API request. The cascade already downloads the janitor's macos-pool-load snapshot for its queued counts, and the snapshot also records each label's oldest_queued_minutes. A label is stalled when either:

  • its oldest queued job has waited 25 min by now (the snapshot's figure plus the snapshot's age), or
  • the snapshot itself saw a wait of 5 min or more and the label has no runner online now (the scale-set runner groups the cascade already reads).

A moving label starts jobs within minutes. Measured on 6 vCPU, 01:04Z to 02:35Z: median 2.6 min, p90 14.7 min. A stalled label gets no job. While the other label still moves, the bot's overflow attempt (retry_runner) avoids the stalled one.

Hosted spill, concurrent with Blacksmith. Order: idle minis, then Blacksmith 12 vCPU, then 6 vCPU, then GitHub-hosted macos-26, then the minis' queue. Hosted takes only the overflow, in both cases:

  • Blacksmith is stalled.
  • Blacksmith is full but moving. Hosted is free on this public repository, so it should not sit idle while Blacksmith queues.

Limits: 5 hosted macOS jobs at once (counted across every hosted macOS pool in the snapshot), and at most 2 per run. New placement outputs hosted_jobs and hosted_runner feed fast-tier and swift-canary's runs-on, and swift-canary's CMUX_NEXT_RUNNER copy of it. The gating matches the Blacksmith slots: attempt 1, or the bot's attempt 2.

What each job pins, and why only two jobs spill:

  • Xcode: fast-tier and swift-canary set no CMUX_CI_XCODE_APP, so select-ci-xcode.sh takes xcode-pins.txt's 26.6. That is the hosted image's default (/Applications/Xcode_26.6.app, build 17F113, the same as Blacksmith).
  • Cache fingerprints: fast-tier-cache-key.sh puts the workspace path in its R2 prefix, so a hosted /Users/runner/work build never restores a Blacksmith seed. It builds cold, which is correct, and gets 45 min instead of 20 when it lands on hosted (timeout-minutes reads the same output).
  • Blacksmith-only steps: neither job uses useblacksmith/* or sticky disks. The owned-only steps are gated on runner.name containing -glaeda.
  • Excluded: the scheme compile and the Release compile stay off hosted. They pin /Applications/Xcode_26.6.app, which the image has. But a cold full app compile with web bundles, the acpmux build and the launch-time budget on about 3 vCPU has not been measured against their 60 min budget. During a stall they keep the minis' queue, which beats a queue that never starts.

Fail toward the previous behavior. If the snapshot or runner groups are unread, no label is stalled and nothing goes to hosted. The placement still runs from the base branch's scripts/ci, so this PR's own CI routes as before.

The hosted-runner guard. tests/test_ci_self_hosted_guard.sh forbids literal GitHub-hosted labels in manaflow-ai workflows. This route names macos-26 only through the placement output, so the guard's literal check still passes. The guard's comment now documents that exception. tests/test_ci_next_macos_cascade.py fails if any job other than fast-tier or swift-canary reads hosted_runner. Nothing checks hosted health: an outage there queues at most the hosted capacity, and a hosted failure falls back only on the bot's attempt 3.

tests/test-execution.toml also gains a tests/test_ios_internal_builder.py entry. The pre-commit hook registered it, and the test exists on the base but was unregistered.

Not in this PR

scripts/ci/simple_pool_picker.py (main's ci.yml) needs the same stall rule for its Blacksmith overflow. Its picker logic is separate, so that is a follow-up into main.

Testing

python3 tests/test_ci_next_macos_cascade.py:

  • At 36fcd28d190, the tests-only commit on base d3a9df0c733: 36 tests, 14 failures and 16 errors.
  • At the head: 37 tests, OK.

Ran on this branch:

  • tests/test_ci_side_lane_placement.py: OK.
  • tests/test_ci_cloud_overflow_switch.py: OK.
  • tests/test_cmux_next_checks_workflow.py: 2 failures, and the same 2 fail on base d3a9df0c733 (test_batch_dispatch_runs_every_dispatch_gated_job, test_every_submodule_free_checkout_on_an_owned_runner_resets_submodules).
  • tests/test_ci_fork_runner_routing.py: 1 failure, the same on base.
  • tests/test_ci_self_hosted_guard.sh: its hosted-runner check passes. It stops at the base's test-ios.yml failure, the same on base.
  • actionlint .github/workflows/cmux-next.yml: clean.

Observed / Not run

Observed: the tests above, run locally with Python 3 and PyYAML. The incident figures come from the placement log of job 118582770774 and the janitor snapshot generated at 10:41:45Z.

Not run: no fast-tier or canary has run on hosted macos-26 yet. The first spill will measure it. Already-queued runs keep their old workflow and placement; the way to move them is in the first comment below.

Risk evidence

Review: SHIP at 51e35133bb79eb3e4187672af18737345e2a7688 (cmux-review subagent; correctness first)
Runtime test: python3 tests/test_ci_next_macos_cascade.py (stall, hosted spill and workflow wiring cases) at 51e35133bb79eb3e4187672af18737345e2a7688 on Huge Blue: OK. PR CI skips its ci guard group for feat-cmux-next PRs (CI fast guards is off for them), and the fleet's ci-step allowlist does not take this script.

Changelog

none

🤖 Generated with Claude Code