PR #32457 - manaflow-ai/cmux

Leo-ask: ask-973a915bea

Leo: "Where the f*** are the generalized solution to all these fing problems that these incessant fing pins?"

Part 1 of 2: the workflows. Part 2 adds the guard that keeps them this way.

Size: about 2,080 changed lines. 932 are one-for-one workflow line swaps (the same transform on 169 workflow files, proven by the round trip below); the rest are the tests that assert those exact lines and must change with them, so it is not split further.

Prior art: #32393 (per-size pools on main) and #32417 (pool copy check) landed the pools these reads use. #27114 (open) moves queued jobs off a stalled label only when their runs-on reads a pool, so this makes it apply to almost every job. #27440 and #27448 (open) change some macOS sizes on lines this touches; whichever lands second rebases.

Why

A runs-on that ends in a literal blacksmith-<N>vcpu-* label can never move: when that size stops starting jobs (2026-10-28, nightly.yml's 4 vCPU fallback), every run waits until someone edits the YAML. One declared runner set fixes that. Each size has its own pool in main's .github/runners.json (#32393), and scripts/ci/runner_pools.py writes each pool's live member to vars.RUNNER_POOLS every 10 minutes. A job that reads its measured size's pool, fromJSON(vars.RUNNER_POOLS).linux_8, follows the pool when it fails over, with no YAML edit. The live variable has had every per-size key since 15:28Z today.

What

Each literal single-size label in a runner selection (runs-on, matrix pools, *RUNNER* env mirrors, reusable-workflow runner inputs) becomes its size's pool:

Literal	Pool
blacksmith-2/4/8/16/32vcpu-ubuntu-2404	linux_2 / linux_4 / linux_8 / linux_16 / linux_32
blacksmith-4vcpu-ubuntu-2404-arm	linux_arm64
blacksmith-6vcpu-macos-15 / -macos-26 / -macos-latest	macos_15 / macos_26 / macos_latest
blacksmith-12vcpu-macos-26	macos_26_large
blacksmith-4vcpu-windows-2025	windows

Kept as they were:

Forks. A fork's pull_request run gets no repository variables. Where a fork branch already exists (head.repo.full_name != github.repository && '<label>'), it keeps its literal and comes before any pool read. Where a workflow a fork can trigger had none, each pool read is wrapped as (<fork pull request> && '<old label>' || fromJSON(vars.RUNNER_POOLS).<pool>), so a fork resolves to exactly the label it resolved to before. Inside a fork branch's own operand, labels stay literal.
Owner checks, membership lists and comparisons (== '<label>', contains(fromJSON('[...]'), ...)) are checks, not routes, and are unchanged.
The trusted-token selector keeps its literal. contains(fromJSON('[...]'), vars.CI_TRUSTED_RUNNER) && vars.CI_TRUSTED_RUNNER || 'blacksmith-4vcpu-ubuntu-2404' (pull_request_target, comment events, CLA, about 27 workflows) is runner trust policy: a pool read there would let whoever writes RUNNER_POOLS route the trusted token. It stays byte-for-byte as on base.
cmux-next-guards.yml signing-runner-variables keeps its literal: it checks the runner variables, so its own route must not read them.
Signing and release jobs keep their routes, byte for byte. A job that holds signing or release material (scripts/ci/runner_routing_table.py's sensitive: Apple/signing/notary secrets, codesign, notarytool, the signing keychain, release uploads) is left as it is. 42 such jobs in 15 workflows keep their routes. A pool read there would let whoever writes RUNNER_POOLS choose the machine that holds the keys. That is runner trust policy, and the release lane's own check (tests/test_ci_release_sdk_lane.sh) holds the release signer's literal fallback whole. Their pickers (pick-sign-runner and the like) hold no secret and do read pools. The light-Linux hosted signer lines in tests/test_ci_self_hosted_guard.sh are unchanged.
No cancellation and no concurrency changes. No cancel-in-progress or concurrency group is touched.

Proof the routing is unchanged: replacing every pool read with the literal it replaced (and unwrapping each fork wrap) turns all 466 changed workflow lines back into their base lines exactly. Every per-size pool's first member is that literal.

Tests that assert Blacksmith is absent (the nightly Blacksmith-only ratchet, the signer never-16/32 checks, the owned-only nightly checks) now also refuse the matching pool read, since every per-size pool is Blacksmith; BLACKSMITH_LINUX_ONLY_DEBT is unchanged from base.

Performance

Each pool's first member is the label the job named before, so routing is unchanged until a size stalls. After that, the job moves to the pool's next member instead of waiting.

Metric	Before	After	Delta	Harness
Blacksmith minutes per day	same jobs, same sizes	same jobs, same sizes	0 while every size starts	`.github/runners.json` pools: first member = previous literal
Jobs that stay queued on a stalled size	every job pinned to it	none (the pool moves within ~10 min)	n/a	`scripts/ci/runner_pools.py` reconcile

Testing

All 147 tests that read workflows or runner routing (tests/ files naming runs-on, RUNNER_POOLS, blacksmith- or the touched workflows) ran on this head, plus tests/test_ci_no_single_size_pins.py:

142 pass.
4 fail exactly as on base 9fde804c, the same tests in the same way: test_check_ghostty_zig_workflows (bashlex not installed), test_ci_cloud_probe_rescue, test_ci_fleet_rule_lints, test_ci_merge_main (needs git 2.46).
test_ci_fast_checks_failover timed out once under the parallel run and passes on its own, as it does on base.
The round trip (each pool read -> its old literal) reads back 466 of 466 changed workflow lines to base, the 2 others being the ci-guards step rename in part 2.

Observed / Not run

Ran all the above locally with python3 and bash. Not run: the workflows themselves on GitHub (routing is unchanged until a pool fails over, by the round trip).

Changelog

none

Agent session: agent session

🤖 Generated with Claude Code