#26360 ci: the overflow switch never uses GitHub-hosted Ubuntu; its watch runs on Blacksmith

Since 19:24Z on 2026-10-27, every ci-cloud-overflow-probe run was cancelled with no jobs. Run 40567582233's watch job waited 94 minutes (19:22Z to 20:56Z) for a GitHub-hosted ubuntu-24.04 runner. That pool was full, partly because the switch's own failover sends LINUX_RUNNER and CI_TRUSTED_RUNNER there. While it waited, the run held the workflow's ci-cloud-overflow-probe concurrency group. GitHub keeps one pending run per group, so each new scheduled run evicted the previous pending one: ten runs were cancelled before getting any jobs, and the switch decided nothing in that time. At 20:56Z the watch finally started, saw the probe had run, and put overflow back on.

Leo, 2026-10-27: "just put it on blacksmith / 16vcpu blacksmiths do exist". No job runs on GitHub-hosted Ubuntu.

"Without it those jobs get zero capacity during an outage, and after it they get a share of the fallback. Without it the switch can never cover them."

Change

• The watch job runs on Blacksmith. While overflow is on (no record), it runs on blacksmith-16vcpu-ubuntu-2404, a size the probe does not ask for (the probe uses blacksmith-4vcpu-ubuntu-2404). While overflow is off (record set), Linux overflow is on that 16 vCPU size, so the watch runs on blacksmith-6vcpu-macos-26; it can then always put overflow back, whatever the Linux sizes are doing.
• Linux failovers are other Blacksmith sizes, never hosted Ubuntu. The switch now writes LINUX_RUNNER = blacksmith-16vcpu-ubuntu-2404 and CI_TRUSTED_RUNNER = blacksmith-2vcpu-ubuntu-2404. CI_TRUSTED_RUNNER's 38 readers only accept ubuntu-24.04 and the 2 and 4 vCPU labels, so 2 vCPU is the only Blacksmith size it can take without workflow edits. CI_CLOUD_FAILOVER may name another Blacksmith Linux size for those two variables and refuses ubuntu-*.
• The stuck-run rescue skips jobs already queued on a failover label. Re-running them would move nothing.
• The macOS failovers (owned pools) are unchanged.
• docs/ci-runners.md's overflow-switch section matches.

Known limit: if every Blacksmith Linux size stalls while overflow is on, the healthy-mode watch on 16 vCPU queues too. That run then holds the concurrency group until Linux returns, and returns are exactly when its probe starts. A switch at that point would only send Linux to another stalled Blacksmith size anyway.

Tests

• tests/test_ci_cloud_overflow_switch.py: the watch's runs-on is pinned to the Blacksmith expression and must not name ubuntu-24.04 or ubuntu-latest. This fails on main, where it is runs-on: ubuntu-24.04. The failover values are covered: no ubuntu-*, and a Blacksmith label only for a Linux variable and never the probe label. Also covered: the override rules and the rescue skipping failover labels.
• tests/test_ci_self_hosted_guard.sh drops the watch's hosted exemption and passes.
• These match main's results: test_ci_repo_variable_defaults, test_ci_fork_runner_routing, test_ci_no_github_hosted_macos, test_runner_label_policy, test_ci_check_repo_variables, test_tui_publish_workflow_security, test_ci_linux_guard_routing.

Agent session: agent session

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Moves the CI overflow switch entirely off GitHub-hosted Ubuntu so its watch never blocks on a full hosted pool again, after it queued 94 minutes on 2026-10-27 and held the workflow's concurrency group while ten later runs were cancelled with nothing decided.

• The watch job now runs on Blacksmith: blacksmith-16vcpu-ubuntu-2404 while overflow is on (a size the probe never asks for), blacksmith-6vcpu-macos-26 while it is off, so it can always put overflow back; it takes macOS whenever the probe itself may ask for 16 vCPU.
• Linux failovers point to other Blacksmith sizes instead of ubuntu-24.04: LINUX_RUNNER to blacksmith-16vcpu-ubuntu-2404, CI_TRUSTED_RUNNER to blacksmith-2vcpu-ubuntu-2404, a size its readers accept. CI_CLOUD_FAILOVER may name another Blacksmith Linux size for those two (trusted only the 2 or 4 vCPU labels) and refuses ubuntu-*.
• A Linux failover never names the stalled probe label itself; when it would, it takes the next Blacksmith size over.
• The stuck-run rescue skips jobs already queued on a failover label that was nobody's value before, since re-running them moves nothing.
• The repo-variable checker flags a CI_TRUSTED_RUNNER whose readers reject it, so a bad value is caught before it silently falls back.
• Records written before this change (Linux failed over to hosted Ubuntu) are migrated to the new Blacksmith failover, keeping the old destination in the record so an interrupted move resumes and still restores. Without the switch token the migration fails loudly instead of leaving Linux on hosted Ubuntu.

Summary by CodeRabbit

• Chores
  • Linux overflow failover now defaults standard and trusted jobs to Blacksmith runners with different runner sizes.
  • Standard Linux failover supports other Blacksmith Ubuntu sizes; trusted-runner overrides are limited to supported sizes. GitHub-hosted Ubuntu overrides are not accepted.
  • Watch jobs use Blacksmith macOS or Linux runners based on overflow status and runner configuration.
  • Jobs already queued on failover destinations are excluded from automatic rescue.
  • Existing failover records targeting GitHub-hosted Ubuntu are migrated to current Blacksmith destinations.
  • Failover selection avoids the stalled probe’s runner size.
  • Repository checks report unsupported trusted-runner settings.