ci(runner pools): LINUX_RUNNER follows the linux pool's live member (4 vCPU queue)

Merged
TE
7d ago
+180-13

Rulebook, "No pins" (Leo, 2026-10-28): "just pin the set of the stuff that isn't hilariously slow." Target: zero queue.

What was wrong

About 250 Linux runs-on on main and feat-cmux-next read vars.LINUX_RUNNER || 'blacksmith-4vcpu-ubuntu-2404'. RUNNER_POOLS.linux is kept live by runner_pools.py (it was blacksmith-32vcpu-ubuntu-2404), but nothing kept LINUX_RUNNER live: it still named blacksmith-4vcpu-ubuntu-2404. That size is outside the declared set and had stopped starting jobs. Measured 2026-10-31 around 03:25Z:

  • all 178 queued jobs in the newest 97 queued runs were on blacksmith-4vcpu-ubuntu-2404;
  • in 50 successful runs created since 02:30Z, 94 jobs started on the 2, 16 and 32 vCPU Linux sizes (median wait 2 to 3 s), and none started on 4 vCPU;
  • the earlier CI-waste audit counted about 520 jobs queued on 4 vCPU, the oldest at 65 minutes.

The overflow probe now probes only the 32 and 16 vCPU sizes (#27121), so it never saw 4 vCPU stall, and test_ci_runner_set.py counts any vars.X as live, so it passed these jobs.

What this changes

  • runner_pools.py writes the linux pool's live member to LINUX_RUNNER on every pass (MIRRORS, mirror_writes()). It reads the value live after the switch step. It leaves the variable alone while the overflow switch's record (CI_CLOUD_OVERFLOW_SAVED) is set, when the variable holds a value that is not a Blacksmith label, and when the read fails.
  • runner_pools.py --check, which ci-repo-variables.yml runs every 15 minutes, now fails while LINUX_RUNNER names a label outside pools.linux and the overflow record is unset. A stalled size outside the set now turns that check red instead of staying hidden.
  • Tests cover: following the pool, leaving it to the switch or to a deliberate value, an unreadable variable, the check, and the workflow passing what the check reads.
  • docs/ci-runners.md "Runner pools" says this.

Done now, by hand, as mitigation (03:38Z)

LINUX_RUNNER is set to blacksmith-32vcpu-ubuntu-2404, the same as RUNNER_POOLS.linux. CI_TRUSTED_RUNNER (unset before, so its jobs used the literal 4 vCPU fallback) is set to blacksmith-2vcpu-ubuntu-2404, which is in its allowed list and starts in seconds. With this PR, the reconciler keeps LINUX_RUNNER on the pool from now on.

Not in this PR (follow-ups)

  • Jobs already queued on 4 vCPU keep that label, because GitHub binds a label when it creates a job. Moving them is the queue janitor's dead-label re-dispatch (#27114, open).
  • The overflow watch job (which runs this reconciler) runs on a literal blacksmith-6vcpu-macos-26. Its run from 02:57Z was still queued 40 minutes later, so the reconciler itself can stall on a Blacksmith macOS label.
  • CI_TRUSTED_RUNNER jobs choose from ["ubuntu-24.04", "blacksmith-2vcpu…", "blacksmith-4vcpu…"] and fall back to a literal 4 vCPU label. They have no pool yet.
  • The roughly 250 vars.LINUX_RUNNER || '<4vcpu>' runs-on should read fromJSON(vars.RUNNER_POOLS).linux directly. That migration needs its own PRs, one per branch.

Risk: low. The reconciler writes one more variable, and only to a declared, ephemeral pool member, which signing variables allow (signing_runner_variables.allowed_value). All the contains([...], vars.LINUX_RUNNER) allowlists on main, and the vars.LINUX_RUNNER == '…16vcpu…' checks on feat-cmux-next/nightly-next, accept 32 and 16 vCPU.

Tests: python3 -m unittest tests.test_ci_runner_pools tests.test_ci_runner_set tests.test_ci_cloud_overflow_switch tests.test_ci_runs_on_values tests.test_ci_check_repo_variables: all pass.

🤖 Generated with Claude Code

Agent session: agent session