#29844 ci: gallery health, web validation, PR checks and the cloud backend check read the live Linux pool, not one Blacksmith size

Leo, 2026-10-31: "I f***ing hate these. Just use one of the existing fleet minis, use AWS, I don't care. Stop falling only to a Blacksmith that isn't available. Stop messing around with holds and waiting for CI that fixes CI." Also: "do we REALLY need to be compiling/testing THAT often...?" and "i assume blacksmith prob cut us off cuz we were using like $10k a day".

What was stuck

At 2026-10-31T23:57Z the cmux-ci controller raised these queue alarms. Each job was queued 44 to 46 minutes on one Blacksmith label, and nothing had started:

job	label	why it resolved there
gallery health / health (4 shards)	blacksmith-16vcpu-ubuntu-2404	!startsWith(vars.LINUX_RUNNER,'blacksmith-') && vars.LINUX_RUNNER || vars.CI_CLOUD_OVERFLOW_SAVED && '32' || '16'. LINUX_RUNNER always names Blacksmith (runner_pools.py mirrors the pool into it), so this reads neither the variable nor the pool. The job waits on one fixed size.
Web validation / web-validation (and build, tests)	blacksmith-32vcpu / 16vcpu	vars.LINUX_RUNNER || '4vcpu' for the small jobs, and the same fixed pair for build and tests
PR checks / base-only update (and size, ui-review, split)	blacksmith-32vcpu-ubuntu-2404	vars.LINUX_RUNNER || 'blacksmith-2vcpu…', a single size (tests/test_ci_linux_single_size.py)
PR cloud backend check	blacksmith-4vcpu-ubuntu-2404	fromJSON(needs.runners.outputs.map).linux. The capability map in runners.json is a static blacksmith-4vcpu-ubuntu-2404, a size that is outside the runner set and that had stopped starting jobs. The map's resolve job is also pinned to 4 vCPU.

The change

• Each of those jobs now reads fromJSON(vars.RUNNER_POOLS).linux after the fork branches. This is the #29757 pattern for ci.yml's changes, app-platform and the terminal crates. runner_pools.py moves the pool off a member that the probe or the janitor saw start nothing, and the janitor's dead-label re-dispatch makes a re-run read the new member.
• pr-cloud-backend-check drops its resolve-runners hop, so each PR event runs one fewer pinned job.
• Tests:
  • tests/test_ci_runner_set.py adds these 11 jobs to its targets. It also fails a static map read or any Blacksmith literal after the fork branches.
  • tests/test_ci_linux_single_size.py gains FIXED_PAIR, a shrink-only ratchet for the fixed-pair shape above (72 jobs in 40 workflows still use it). A new one fails.
  • tests/test_ci_linux_compile_runners.py drops the three entries that moved.

What this does not do yet (fleet first)

• GCP fleet Linux is not live. HQ's scaler (cmuxterm-hq#5561, label cmux-fleet-linux-x64) logs "cmux-gha-runners workload identity exchange failed (grant not in place: build-fleet/swift-linux-builder/gha-runners/grant.sh); no runner VM made" every 5 min. A GCP project owner has to run grant.sh once. After that, these jobs can take the label first, and HQ's gha-runners/workflows.json must list these workflows.
• The AWS Mac pools (glaeda-aws-*) have no runners. runner_label_policy.py and pr_runner_pool.py say so.
• The owned minis were saturated. glaeda-std-xcode-26.6 had 1h41m of queue, and the side label "runs nothing" (NOW). portable_job_placement.py already sends portable jobs to an idle mini. These jobs need apt (Playwright --with-deps) or a docker service (database), or they are tiny status jobs where a placement hop would cost more than the job.
• The Release compile (Xcode 26) on blacksmith-6vcpu-macos-26 is unchanged. Its runs-on already goes through macos-placement (owned Macs first, stall-aware Blacksmith sizes). It fell to Blacksmith because no owned Mac was free, so that is a capacity problem, not a pin.

Performance

metric	before	after	delta	harness or run link
jobs queued on a Blacksmith size that started nothing (these 11)	44-46 min (2026-10-31T23:57Z alarms)	the pool moves off a dead member within one runner_pools.py pass (about 10 min), then the janitor re-runs	not measured yet	NOW.md queue alarms
Blacksmith jobs per cloud-backend PR event	2 (resolve on 4 vCPU, plus the check)	1	-1 job (about 20 s on 4 vCPU)	this diff
Blacksmith vCPU-minutes per day (these 11 jobs)	small jobs on the pool's 32 vCPU through the LINUX_RUNNER mirror; gallery health 4 shards and web-validation tests on fixed 16 vCPU	small jobs unchanged; gallery health and web-validation tests take the pool member (32 first, 16 second)	about neutral; up to 2x vCPU rate for those 5 legs while the pool sits on 32 (no per-job minute source readable from this lane VM)	runners.json pools.linux on main

Risk: CI routing only. No step changes. A fork PR keeps its own Blacksmith label from the fork branch.

Agent session: agent session

🤖 Generated with Claude Code