PR #31884 - manaflow-ai/cmux

What

RUNNER_POOLS.linux and LINUX_RUNNER both named blacksmith-4vcpu-ubuntu-2404. About 100 runs-on read the pool and about 285 read LINUX_RUNNER. On 2026-11-02 at about 04:40Z that size had 428 jobs queued beside 535 running, with the oldest waiting about 45 minutes (721 queued earlier). At the same time blacksmith-16vcpu-ubuntu-2404 had about 11 queued. At 05:25Z it was still 179 queued with the oldest at 2,712 s, while 16 vCPU had 1 queued (fleet dashboard pools). Main's scripts/ci/runner_pools.py moves a pool only off a dead member. A size that keeps starting jobs is never dead, however long its queue, so nothing moved.

scripts/ci/linux_pool_spill.py is a feat-side entry point that main's overflow watch runs beside main's unchanged policy. It follows the same pattern as runner_pool_reconcile.py (#30714). Main takes only workflow wiring, in a follow-up main PR that pins this commit. Each pass:

Runs main's runner_pools.main() unchanged, through a switch that records writes instead of making them. The dead-member rule, the probe and janitor evidence, and the --check schema all stay main's.
Samples the Linux queue: the newest 60 queued and in-progress runs from the last 6 hours. It reads their jobs with the job's own GITHUB_TOKEN (the watch already has actions: write), which is at most 62 reads per pass. It counts queued, running and waited jobs per Blacksmith Linux label, where waited means queued 3+ minutes (the stall-aware failover's bound). This is a lower bound.
Spill: when 15+ sampled jobs on the linux pool's live member have waited, and a later live member has at most 3, the pool moves there. The move is recorded in RUNNER_POOLS._spill.
Come back (hysteresis), on whichever comes first:
- after 20+ minutes, on a complete sample showing the member drained (nothing waited, at most 5 queued, at most 70% of what it ran when the pool left);
- on a complete sample where the target backs up and the member it left does not;
- after 60 minutes in any case. If the member is still backed up, the next pass spills again, so a spill never outlives the backlog by much.
Writes RUNNER_POOLS and the LINUX_RUNNER mirror only on a change. The mirror goes through main's mirror_writes(), so the overflow switch's record still wins.

An unread sample never moves the pool. A spill target must be a live member of main's declared pool (4 then 16 vCPU), so a spill never leaves the runner set. Main's tests already check that every bounded vars.LINUX_RUNNER selector accepts 16 vCPU.
It moves no queued job. Jobs already bound to 4 vCPU drain there.
No new credential, secret, permission or runner-trust change. It uses the watch job's existing route-App token for variable writes and GITHUB_TOKEN for reads.

This replaces #31861, which put the same logic into main's policy. That PR is refused: main takes only CI wiring and data files.

Performance

metric	before	after	delta	harness or run link
4 vCPU Linux queue, oldest wait	~45 min (04:40Z), 2,712 s (05:25Z)	target < 5 min within ~20 min of the first spill	—	fleet dashboard `pools`, measured after the main wiring lands
Blacksmith job-minutes/day	unchanged	unchanged (the same jobs, on a bigger size while spilled)	~0	—
Blacksmith vCPU-minutes/day (estimate)	baseline	+ spilled job-minutes × 3 (16 vs 4 vCPU). E.g. 3 h/day of spill at ~150 jobs arriving at once is ~27k job-minutes/day on 16 vCPU, about +81k 4 vCPU-equivalent minutes/day. This is bounded by the backlog plus at most 60 min per spill	+ only during backlog	estimate; replace with measured per-label minutes after a day
API reads per watch pass	0 for the queue	≤ 62 GITHUB_TOKEN reads (every 10 min, ≤ ~372/h)	+≤62	`test_the_sample_is_bounded`

No timer, observer or loop is added. The script runs once per existing watch pass and stops.

Validation

RUNNER_POOL_POLICY_ROOT=<main checkout at 4ac6e872ac1> python3 -m unittest tests.test_ci_linux_pool_spill: 15 OK. Covers:
- spill rules: spill; no spill on a short queue, a busy or dead sibling, or no sample; hold; an unread sample keeps the spill; drain only on a complete sample after the hold; target backs up; max hold; dead target or new first member;
- sampling: counts, partial and unread samples, the call bound;
- end to end with main's unchanged policy: spill and LINUX_RUNNER mirror, main's --check accepts the result, a steady pass writes nothing although main's policy alone would flip back, return on drain, a dead member stays main's call, dry run.
tests.test_ci_runner_pool_reconcile with the same policy root: OK. tests/test_ci_test_execution_registry.py: 34 OK (the new test is registered in the manual lane, so no new CI gate).

Observed / Not run

Ran the suites above. Not run: a live watch pass. That happens once the main wiring PR lands.

Changelog

none

Agent session: agent session

🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

Adds a queue-aware spill for the Linux runner pool so CI moves off a backed-up size instead of only a dead one. Main's runner_pools.py only moves a pool off a dead member, which never triggers when a size keeps starting jobs despite a long queue.

scripts/ci/linux_pool_spill.py runs main's unchanged policy through a recording switch, then samples the newest 60 in-flight runs per Blacksmith Linux label and spills linux to a later live member (4 → 16 vCPU) when 15+ sampled jobs waited 3+ minutes and the target has at most 3. It returns after a complete-sample drain (20+ minutes held), when the target backs up, or after 60 minutes in any case. LINUX_RUNNER follows through main's mirror_writes(). No queued jobs are moved, no new credentials are needed, and an unreadable sample never moves the pool.

A run showing a full page of 100 jobs marks the sample incomplete, so a drain is never proven from a partial read.
RUNNER_POOLS is re-read before writing, so a value another writer changed during the pass is not overwritten.
Replaces #31861, which put the same logic into main's policy: this PR only adds the feat-side entry point and tests, leaving main's wiring for a follow-up PR.
Tests are registered in the manual lane; the end-to-end cases need RUNNER_POOL_POLICY_ROOT pointing at a main checkout.