ci: fail over hard-coded Blacksmith Linux runs-on while the overflow record is set

Closed
TE
9d ago
+198-63

Since about 18:28Z on 2026-10-27 Blacksmith Linux (every ubuntu-2404 size) has started no jobs for manaflow-ai. The overflow switch fired at 18:42Z: it wrote CI_CLOUD_OVERFLOW_SAVED and pointed LINUX_RUNNER / CI_TRUSTED_RUNNER at ubuntu-24.04. But jobs whose runs-on names a Blacksmith Linux label literally read neither variable, so they got zero capacity for the whole outage (the fast-checks run has been queued since 18:32Z).

"Without it those jobs get zero capacity during an outage, and after it they get a share of the fallback. Without it the switch can never cover them."

Change

Every such runner label in a non-excluded workflow now puts the record ahead of its final Blacksmith Linux literal:

  • before: runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || 'blacksmith-32vcpu-ubuntu-2404' }}
  • after: runs-on: ${{ github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04' || vars.CI_CLOUD_OVERFLOW_SAVED && 'ubuntu-24.04' || 'blacksmith-32vcpu-ubuntu-2404' }}
  • a bare runs-on: blacksmith-... becomes ${{ vars.CI_CLOUD_OVERFLOW_SAVED && 'ubuntu-24.04' || 'blacksmith-...' }}

While the record is set these jobs take the hosted ubuntu-24.04 fallback (ephemeral, so the ephemeral-required jobs stay ephemeral). Healthy routing is unchanged: an unset variable is falsy, so the Blacksmith literal still wins.

60 runner lines in 23 workflows. Also covered: cmux-tui runner:/linux_runner: (the term goes after the fork pull-request branch, which still picks Blacksmith), cmux-tui-artifacts DAEMON_POOL (kept equal to the daemon-test job's runs-on), the swift-canary runs-on/CMUX_NEXT_RUNNER pass branch (pass && overflow && 'ubuntu-24.04' || pass && 'blacksmith-2vcpu-ubuntu-2404', so the fork-routing parser still sees a guarded literal), contains(vars.LINUX_RUNNER, '32vcpu') and vars.CI_COMPILE_CHECK_GATEWAY selectors (neither follows the switch). One bare cmux-next.yml runs-on also gets the usual repository_owner hosted branch, which clears one of the four base failures in tests/test_ci_fork_runner_routing.py.

Left alone: lines that already read vars.LINUX_RUNNER / vars.CI_TRUSTED_RUNNER, ARM/macOS/Windows labels, runner.name checks, the overflow probe itself, the Blacksmith Testbox setup workflow, and nightly, notarization, dev-daily and release/publish/TestFlight/App Store/Homebrew workflows (sequenced separately).

Tests

  • New tests/test_ci_blacksmith_linux_failover.py: fails on a non-excluded runner line whose final Blacksmith Linux literal has no failover (60 offenders on base, 0 here).
  • tests/test_cmux_next_checks_workflow.py: its "reads a variable, so may reach a mini" proxy ignores the failover term (it only ever picks hosted ubuntu-24.04).
  • Passing locally: the two above, test_ci_runs_on_fallback, test_ci_linux_compile_runners, test_ci_next_macos_cascade, test_ci_side_lane_placement, test_ci_compile_test_plan, test_cmux_tui_daemon_tests_gate, test_ci_cloud_overflow_switch, test_ci_check_repo_variables, test_ci_repo_variable_defaults. test_ci_fork_runner_routing (3 failures, was 4) and test_runner_label_policy (2) fail identically on base.
  • docs/ci-runners.md overflow-switch section says what these lines do.

Agent session: agent session

šŸ¤– Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes overflow failover for jobs that hard-code a Blacksmith Linux runs-on label. During the 2026-10-27 Blacksmith outage those jobs queued with zero capacity for the whole event because they read neither LINUX_RUNNER nor CI_TRUSTED_RUNNER; while the overflow record is set they now fall back to hosted ubuntu-24.04 (ephemeral, so ephemeral-required jobs stay ephemeral), and healthy routing is unchanged since an unset record is falsy.

  • Puts vars.CI_CLOUD_OVERFLOW_SAVED && 'ubuntu-24.04' || before every Blacksmith Linux literal in 60 runner lines across 23 workflows; the test guards all literals on a line, not only the final one.
  • Covers cmux-tui runner:/linux_runner:, cmux-tui-artifacts DAEMON_POOL, the swift-canary CMUX_NEXT_RUNNER pass branch, and web-complexity's every-PR Blacksmith branch.
  • Leaves lines already reading vars.LINUX_RUNNER/vars.CI_TRUSTED_RUNNER and non-Linux labels unchanged; nightly, notarization, dev-daily, and release/publish workflows are excluded from the guard.

Tests
tests/test_ci_blacksmith_linux_failover.py fails on any non-excluded runner line without failover (60 offenders on base, 0 here); the test_cmux_next_checks_workflow.py proxy ignores the failover term so it still only ever picks the hosted label.