ci(runner pools): spill the linux pool off a backed-up size, not only a dead one
What
scripts/ci/runner_pools.py (run by ci-cloud-overflow-probe.yml's watch every 10 minutes, from main) moved a pool only off a member it judged dead. A size that keeps starting jobs is never dead, however long its queue. On 2026-11-02 at about 04:40Z blacksmith-4vcpu-ubuntu-2404, which RUNNER_POOLS.linux and LINUX_RUNNER both pointed at (about 100 runs-on read the pool and about 285 read LINUX_RUNNER), had 428 jobs queued beside 535 running, the oldest about 45 minutes (721 queued earlier). 16 vCPU had about 11 queued at the same time. Nothing moved. At 05:25Z it still showed 179 queued, oldest 2,712 s, while 16 vCPU had 1 queued.
Now the linux pool is queue-aware:
queue_janitor.py: themacos-pool-loadsnapshot gainslinux_load: per Blacksmith Linux label,queued,running, andwaited(queued 3 minutes or more, the stall-aware failover's bound). It comes from the runs the sweep already lists, so it costs no extra API calls. It is a lower bound, because Linux-only workflows' runs are not listed (needs_jobs()). The macOSpoolsmap the pickers read is unchanged.runner_pools.pyspill_choice()(only forQUEUE_AWARE_POOLS = ("linux",)):- Spill: when 25 or more jobs on the pool's first live member have waited 3+ minutes and a later live member has at most 3 such jobs, the pool moves there. The move is recorded in
RUNNER_POOLS._spill(from,to,since, andrunningat the time of the move). - Come back (hysteresis): after at least 30 minutes, and only once the member it left has nothing waited, at most 5 queued, and at most 70% of the jobs it was running at the move. It also comes back earlier if the target backs up (25+ waited) while the member it left does not.
- A dead target, or a dead first member, ends the spill. The existing dead rule then decides.
- Unknown load never moves the pool. That covers no snapshot, one older than 20 minutes, a truncated one, or one without
linux_load. An existing spill is kept as it is. LINUX_RUNNERfollows the pool through the existing mirror (mirror_writes()). The bounded selectors accept 16 vCPU (test_every_bounded_selector_accepts_every_linux_member).
- Spill: when 25 or more jobs on the pool's first live member have waited 3+ minutes and a later live member has at most 3 such jobs, the pool moves there. The move is recorded in
- Each pass is bounded and idempotent: a steady state writes nothing (test), and a malformed
_spillis dropped and repaired. - Also fixes
test_a_macos_only_overflow_record_does_not_freeze_linux_runner_on_a_dead_member, which was red on main. It still used the old 32/16 members, but the pool has been 4/16 since 11-01.
Data used: the fleet dashboard (pools: per-label queued, running and oldest wait) for the incident numbers. The live decision uses only the janitor snapshot the reconciler already reads. No new credential, secret, permission or runner-trust change. It moves no queued job: jobs already bound to 4 vCPU drain there as new arrivals go to 16.
Base is main because the overflow watch and the queue janitor are scheduled workflows and run main's copies.
Performance
| metric | before | after | delta | harness or run link |
|---|---|---|---|---|
| 4 vCPU Linux queue, oldest wait | ~45 min (04:40Z), 2,712 s (05:25Z) | target < 5 min within ~20 min of the first spill (two janitor and watch passes) | — | fleet dashboard pools; measured after merge in a follow-up comment |
| 4 vCPU Linux queued | 428 (04:40Z), 179 (05:25Z) | drains as arrivals go to 16 vCPU | — | same |
| Blacksmith job-minutes/day | unchanged | unchanged (the same jobs run, on a bigger size while spilled) | ~0 | — |
| Blacksmith vCPU-minutes/day (estimate) | baseline | + spilled minutes × 3 (16 vs 4 vCPU). E.g. 3 h/day of spill at ~150 jobs arriving at once is ~27k job-minutes/day on 16 vCPU instead of 4, so ~+81k 4 vCPU-equivalent minutes/day. This is bounded by the spill window and ends when 4 vCPU drains | + only during backlog | estimate; to be replaced with the measured per-label minutes after a day |
Cost/latency trade: a spill pays 4× per minute for jobs that would otherwise wait 10 to 45+ minutes. It happens only while 4 vCPU is measurably backed up and 16 vCPU has room, and it reverts on its own.
Validation
python3 -m unittest tests.test_ci_runner_pools: 51 OK, including 13 new spill tests (spill, no spill on a short queue or a busy or dead sibling, unknown load, hold then drain, steady state writes nothing, target backs up, dead target or member, non-queue-aware pools, malformed records, freshness, andmain()end to end with theLINUX_RUNNERmirror and--check).python3 -m unittest tests.test_ci_queue_janitor: 338 OK, including the newlinux_loadtests.- Passing: cloud overflow switch, runner set, simple pool picker, owned pool rescue, runner label policy, check repo variables, fast checks failover and repo variable defaults.
tests.test_ci_pr_runner_pool'sIOSWiring.test_the_simulator_build_holds_no_secret...fails the same way on main without this change.
Agent session: agent session
🤖 Generated with Claude Code
![View with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/view-with-codesmith-dark-v3-3x.png)
![Autofix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-dark-3x.png)
Need help on this PR? Tag @codesmith-bot with what you need.
Summary by cubic
Fixes the linux runner pool so it can move off a backed-up member, not only a dead one. A size that keeps starting jobs was never considered dead, so on 2026-11-02 the 4 vCPU pool sat with 428 jobs queued beside 535 running (oldest about 45 minutes) while 16 vCPU had room and nothing moved.
The linux pool is now queue-aware: the janitor snapshot adds linux_load (per-label queued, running, and jobs waited 3+ minutes, computed from runs already listed, so no extra API calls), and spill_choice() moves the pool to a later live member when 25+ jobs on its member have waited 3+ minutes and that member has at most 3. The move is recorded in RUNNER_POOLS._spill; the pool comes back after 30 minutes once the member it left has nothing waiting, at most 5 queued, and runs at most 70% of what it ran at the move, or sooner if the target backs up. A dead member ends the spill, unknown or stale load never moves the pool, and each pass is idempotent. LINUX_RUNNER follows the pool through the existing mirror.
Also fixes test_a_macos_only_overflow_record_does_not_freeze_linux_runner_on_a_dead_member, which was red on main because it still used the old 32/16 members.
Summary by CodeRabbit
- New Features
- Linux CI jobs can move from a backed-up runner pool member to a later member with more capacity, helping reduce queue delays. Jobs can return to the original member when demand eases.
- Documentation
- Updated the CI runner guide with details about Linux queue-aware routing and when jobs move between pool members.