PR #30878 - manaflow-ai/cmux

Leo, 2026-11-01: "ok. we clearly are not able to use them. so we should have resized a long-ass time ago."

The case

Run 41299099061 (cmux-tui spec inventory, for #30775) has had five jobs queued since 10:48Z while its one 8 vCPU job ran. inventory and rustfmt took vars.LINUX_RUNNER || 'blacksmith-4vcpu-ubuntu-2404', so they waited on a size that had stopped starting jobs. Across the workflows, two runs-on tails keep sending jobs to one Blacksmith size after it stalls:

vars.LINUX_RUNNER || 'blacksmith-<n>vcpu-ubuntu-2404': one variable holding one size.
the hard-coded overflow pick, vars.CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-8vcpu-...' || 'blacksmith-4vcpu-...': while the overflow record is set, every such job goes to the size written in the expression, whether or not that size is live.

Change

60 runs-on lines in 46 workflows now end ... || fromJSON(vars.RUNNER_POOLS).linux }}, after the existing owner and fork branches. This is the same form #30385 (web-subarea-scope) and #30685 (backend) used. A line that had no fork pull-request branch gets one, naming the job's previous size, which test_ci_repo_variable_defaults requires. The lines changed are:

58 lines whose tail was a bare vars.LINUX_RUNNER || 'blacksmith-4vcpu|2vcpu-...' (LINUX_RUNNER already mirrors the pool, so no job changes size);
2 lines whose tail was an overflow pick around a 4 vCPU size (the pool's first member). Any !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER term on those lines goes too, because the pool already carries any fleet member.

There is no literal after the pool read. The repo decided against one (test_ci_repo_variable_defaults, "RUNNER_POOLS has no literal fallback on purpose"): an unset RUNNER_POOLS fails the run at once, a fork pull request never reaches the read, and test_ci_blacksmith_linux_failover would flag a trailing Blacksmith literal that no failover guards.

scripts/ci/owned_pool_rescue.py: the 35 moved workflows are added to feat-cmux-next's UNRESCUED_WORKFLOW_PATHS acknowledgement copy (the pattern from #30867; no behaviour change), because test_ci_owned_pool_sources counts a RUNNER_POOLS read as a possible owned route.

.github/runners.json linux_sizes: cmux-tui.yml#msrv-check now says "pool": "linux" in place of its failover, and the table comment explains that key.

Left as is, and why

lines	why
78	the job reads `secrets.` or has an `environment:`. RUNNER_POOLS.linux may be the secretless fleet route (runner_pool_reconcile.py keeps `linux_ephemeral` for secret-bearing jobs; tests/test_ci_runner_pool_reconcile.py). These keep the ephemeral `LINUX_RUNNER` mirror.
40	the job's permissions grant a write token. Same reason.
22	the job feeds a credentialed job through `needs`: its outputs or its success decide what a secrets, environment or write-token job signs, publishes, pushes or runs (e.g. stable-next `plan`, cloud-vm-migrate `preflight`, pr-autofix `repair`). Same reason.
10	marked `ephemeral-required` (holds a write token, reads no variable)
44	compile jobs with a measured size and outage switch that tests/test_ci_linux_compile_runners.py pins. This includes cmux-tui-spec's acpmux-tests, acpmux-protocol-ir, clippy and api-compat.
15 + 8	measured 2 or 8 vCPU picks: not the pool's size (main's `pools.linux` says a job that needs more cores names its size)
3	portable-placement tails (their own ordered contract)
1	another variable picks first (fleet-test-compile `CI_TEST_COMPILE_GATEWAY`)
all	release and publish workflows (release, publish, testflight, app-store, homebrew, bootstrap: the name set of test_ci_blacksmith_linux_failover) and the reusable workflows they call (cmux-next-source-archive, ios-screenshots, resolve-dispatch-ref). Their build and verify jobs make or check artifacts that get published, so moving them onto a pool that may become the GCP fleet route is a supply-chain trust change and not a routing one. Also nightly and notarization workflows (they never use Blacksmith; test_ci_nightly_off_blacksmith), macOS selectors, and files open PRs already change: gallery-health, pr-checks, pr-cloud-backend-check and web-validation (#29844), cmux-tui-artifacts and cmux-tui-build-package (#30622/#30724), ci-web (#29981), merge-group-policy-checks and cmux-next-web (their text is pinned by tests)

What this does not fix: the measured 8 vCPU compile jobs in the stuck run still fail over to the 4 vCPU size while the overflow record is set. The pool can only move these jobs once its writer has a live member to move them to. Main's pools.linux is [4, 16], both of which are stalled now, and with every member marked the pool keeps its first member. That is the pool writer's lane (scripts/ci), which this PR does not touch.

Test

New: tests/test_ci_overflow_pick_reads_pool.py (lane linux-guard). It counts, per workflow, the Linux jobs that still pick one Blacksmith size through CI_CLOUD_OVERFLOW_SAVED && 'blacksmith-…' or a bare vars.LINUX_RUNNER || 'blacksmith-…', and fails a workflow whose count grows. A count may fall without an edit, so a PR that moves a job never turns another PR's base red. It also checks that cmux-tui-spec's inventory and rustfmt read the pool. This follows the routing-guard precedent of test_ci_linux_single_size (#30360), although cmux CLAUDE.md says not to add CI tests. test_ci_linux_single_size SINGLE_SIZE counts drop to what remains.

Red on the base workflows: python3 tests/test_ci_overflow_pick_reads_pool.py reports 3 failures (inventory, rustfmt, counts grown). Green on the head: OK.
227 files run one by one (tests/test_ci_*.py, test_runner_*, test_runs_on*, the nightly and Linux routing tests and test_web_validation). The pass/fail set is the same as base, plus the new file. The 18 files red on base are still red on the head: test_ci_blacksmith_linux_failover, test_ci_pr_runner_pool, test_ci_signing_keychain_is_deleted_before_bundle_code and others. test_ci_canonical_build_root flips between runs.
python3 scripts/verify-local.py: localization and package-groups fail, and fail the same way on base. Everything else passes.

Observed / Not run

Observed: the test runs above, and YAML parse of every changed workflow. Follow-ups: #31004.

Not run: a live dispatch of a changed workflow, actionlint (not installed on this host).

Performance

metric	before	after	delta	harness
Blacksmith minutes/day	n/a	n/a	routing only	no job changes size. The 58 bare-LINUX_RUNNER jobs already took the pool's member through the mirror. The 2 overflow-pick jobs keep 4 vCPU while it is live and take the pool's next member (16 vCPU, or fleet Linux once the writer adds it) instead of 8 during an outage

Changelog

none

Agent session: agent session

🤖 Generated with Claude Code