ci: hard-coded Blacksmith Linux runs-on fail over to another Blacksmith size
Leo: "just put it on blacksmith / 16vcpu blacksmiths do exist"
This replaces #26339, whose overflow failover went to GitHub-hosted ubuntu-24.04. It now goes only to Blacksmith, using the same size pairing as the overflow switch from #26360 (LINUX_RUNNER 16 to 32 vCPU, CI_TRUSTED_RUNNER 2 to 4).
When Blacksmith Linux starts nothing, the switch writes the record CI_CLOUD_OVERFLOW_SAVED and repoints LINUX_RUNNER and CI_TRUSTED_RUNNER. A runs-on that names a Blacksmith Linux label literally never saw that, so its job sat queued for the whole outage on 2026-10-27.
| runs-on shape | before | while the record is set (after) |
|---|---|---|
| 61 literal Blacksmith Linux labels across 24 workflows | the label, with no failover | another Blacksmith size: 2 vCPU to 4 (27 lines), 4 to 16 (14), 16 to 32 (4), 32 to 16 (15) |
60 compile selectors !startsWith(vars.LINUX_RUNNER, 'blacksmith-') && vars.LINUX_RUNNER || '<16 or 32>' | the switch's Blacksmith value never moved them | the other size (16 and 32 swap). An operator's non-Blacksmith LINUX_RUNNER still comes first. |
| anything | never GitHub-hosted Ubuntu |
An unset record is falsy, so routing is unchanged while Blacksmith is healthy. Fork branches and fork pull-request branches stay first. The write-token jobs (ephemeral-required) still land only on ephemeral Blacksmith VMs.
About the GCP build pool: it has no runs-on label. Its VMs are build-controller workers, reached through cmux-ci submit from a job that is already running. So it can't be a runs-on fallback, and a different Blacksmith size takes that role.
Tests:
tests/test_ci_blacksmith_linux_failover.pyis new, registered intests/test-execution.toml, and runs in ci-guards. It fails on:- a Blacksmith Linux literal without its failover;
- a hosted failover;
- a failover to the same size;
- a lone failover with no primary.
- Updated for the new expressions:
test_ci_linux_compile_runners.py,test_cmux_tui_headless_debug_workflow.py,test_cmux_next_generated_catch_up.py,test_cmux_next_checks_workflow.pyandtest_supplychain_security.py. - 792 routing and guard tests on the 32 vCPU Linux fast-checks runner fail the same 5 on base and head (fork-routing and runner-variable report), so this PR adds no failures. The supplychain check on nightly.yml's
pick-nightly-sign-runneris also red on base and is outside this diff. - Nightly, notarization, dev-daily and release or publish workflows are not covered here, as in #26339.
Risk: low. Each change takes effect only while the overflow record exists, and it trades a stalled Blacksmith size for another Blacksmith size.
Agent session: agent session
š¤ Generated with Claude Code