ci(runner pools): an overflow record defers only the variables it changed, so LINUX_RUNNER leaves a dead size
Leo, on the Blacksmith 32 vCPU queue: "ok. we clearly are not able to use them. so we should have resized a long-ass time ago."
Leo: "(im pretty sure blacksmith trimmed us off a bit just cuz we were allegedly costing like $10k a day) so we need to figure out solulus"
What was wrong
runner_pools.py mirrors the linux pool's live member into LINUX_RUNNER, which about 250 Linux runs-on read as vars.LINUX_RUNNER || .... It skipped that mirror whenever the overflow switch's record (CI_CLOUD_OVERFLOW_SAVED) existed at all.
On 2026-11-01 that record was set at 04:23Z, and its changed names only macOS variables. At 05:30Z RUNNER_POOLS marked blacksmith-32vcpu-ubuntu-2404 _dead and moved linux to 16 vCPU, but LINUX_RUNNER stayed on 32. A sweep at 06:20Z counted 751 jobs queued on 32 with no runner across the 390 runs the broker let it list. Extrapolated to every queued run, that is about 1,900. They include 2 vCPU-class one-file checks (PR perf section, capture, design spec) that wait on 32 only because of this variable.
Change
switch_owned(record)returns the MIRRORS variables the record'schangednames. The mirror and--checknow defer only those variables to the switch.- A record whose
changedcan't be read still defers every variable, as before. - The switch only puts back the variables it recorded ("unless someone changed that variable since"), so it never fights this.
- Regression test: a macOS-only record with 32 marked dead now writes
LINUX_RUNNER=blacksmith-16vcpu-ubuntu-2404. A record that names LINUX_RUNNER, or one that can't be parsed, still writes nothing.
Tests
python3 -m unittest tests.test_ci_runner_pools: 36 pass. On the oldrunner_pools.pythe new test and the updated--checkcase fail (2 failures).test_ci_cloud_overflow_switch,test_ci_check_repo_variables,test_ci_pr_runner_pool,test_ci_runner_set,test_ci_simple_pool_picker,test_ci_fast_checks_failover,test_classic_nightly_build_runner,test_runner_label_policy: OK.
Performance
Blacksmith minutes and cost per day come from HQ build-fleet/data/raw speed-raw (billed minutes, rounded up per job). Prices come from blacksmith runners catalog as quoted in cmuxterm-hq build-fleet/swift-linux-builder/blacksmith/RESULTS.md. The 16 vCPU price is estimated linearly ($0.032/min).
| metric | before | after | delta | harness or run link |
|---|---|---|---|---|
| jobs queued on 32 vCPU (cmux) | 751 listed, ~1,900 estimated (06:20Z) | to be measured after the next runner-pools pass and re-runs | broker sweep, /tmp/fleet1-bs32-queue.md | |
| Blacksmith billed min/day, all sizes | 499,871 (10-31) | expected about the same (this moves jobs between sizes, it does not remove them) | ~0 | speed-raw 2026-10-31 |
| Blacksmith $/day | ~$19.3k (10-31), of which 32 vCPU ~$12.4k | jobs reading LINUX_RUNNER bill at $0.032/min instead of $0.064/min | up to -50% on the moved jobs' minutes | same |
This PR does not cut Blacksmith minutes by itself. It stops jobs from waiting on a size that starts nothing, and it puts them on the smaller live size the pool already picked. Fleet-first routing and per-job sizes are the other fleet1 PRs. Nothing is skipped here: no compile is added or removed.
No new timer, observer or loop: this runs inside the existing 10-minute ci-cloud-overflow-probe.yml watch pass.
Agent session: agent session
🤖 Generated with Claude Code
![View with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/view-with-codesmith-dark-v3-3x.png)
![Autofix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-dark-3x.png)
Need help on this PR? Tag @codesmith-bot with what you need.
Summary by cubic
Fixes runner_pools.py so an overflow record only defers the mirror variables it actually changed, instead of freezing all mirrors. On 2026-11-01 a record that changed only macOS variables kept LINUX_RUNNER on a dead 32 vCPU runner while the pool moved to 16 vCPU, leaving about 1,900 jobs queued on a size with no runners.
Changes
switch_owned(record)returns the variables named in the record'schanged; the mirror and--checkskip only those, and unreadable records still defer everything as before.- Adds a regression test for macOS-only records, records that name
LINUX_RUNNER, and unreadable records.
Summary by CodeRabbit
- Bug Fixes
- Runner pool checks and updates now distinguish which platform was affected by an overflow change. Changes limited to macOS no longer prevent eligible Linux checks or updates, while Linux changes continue to defer them. If overflow information cannot be read, checks and updates remain deferred for all platforms.