PR #27129 - manaflow-ai/cmux

Leo, 2026-10-28: "Bro, I thought we already had logic to automatically route stuff. How can we still have pins like this? I hate pins. Like, just pin the set of the stuff that isn't hilariously slow. Like most of what GitHub has is hilariously slow. We can't be using them."

And, on the pins that stayed: "Where the f*** are the generalized solution to all these fing problems that these incessant fing pins?"

Why

Every workflow writes its own runner label today: about 600 runs-on lines name a literal Blacksmith size, a hosted image or a single label, usually as the fallback after ||. When one label stops starting jobs, nothing moves those jobs: the overflow switch probes one label (4 vCPU) and flips a dozen variables, and a literal after || never reads them. On 2026-10-28 the 16 vCPU Linux label started nothing from about 22:12Z while 32 vCPU served, and about 1,300 runs sat queued. Tonight a nightly signer job sat on a dead 4 vCPU fallback the same way.

What (this PR: the routing source and its reconciler; no job reads it yet)

One declaration. .github/runners.json gets pools: each class of work (linux, linux_arm64, windows, macos_15, macos_26, macos_26_large, provenance, macos_intel) with its members in preference order. linux is the runner set: Blacksmith 32 vCPU, then 16 vCPU. Every member is an ephemeral Blacksmith label, so a job reading a pool never lands on a reused machine whoever started the run; owned Macs keep coming first through the existing pickers' outputs, and the pool is what follows them. Only provenance (npm/PyPI provenance and attestations accept only GitHub's OIDC) and macos_intel (Blacksmith has no Intel Mac) may name a hosted image, each with its reason. Signing jobs are not routed through a pool: they keep their literal ephemeral labels and move through the stall-aware signing picker (#26944, #27113, #27066).
One variable. RUNNER_POOLS holds each pool's live member as JSON. Jobs will read fromJSON(vars.RUNNER_POOLS).<pool>; fromJSON of an unset variable fails the workflow at once, so a missing route is a red run instead of a job queued forever.
One writer. scripts/ci/runner_pools.py, run by the overflow probe's watch job every 10 minutes after the switch, with the switch token it already mints (manaflow-glaeda-route; no new credential or permission). It only chooses. Liveness comes from per-label start evidence: the switch's per-label health CI_CLOUD_LABEL_HEALTH (#27121 probes every Linux member, 32 and 16 vCPU, never the 4 vCPU label), read live through the API; and the janitor snapshot's pools (a Blacksmith label whose oldest queued job waited 10 min, simple_pool_picker.blacksmith_stalled()) plus dead_labels once #27114 lands. The snapshot's starved/unserved describe owned glaeda-* labels, never pool members, so they are not read. It points each pool at its first unmarked member and writes only on a change. A mark expires after 30 min unless seen again; a label seen starting a job clears at once. With every member marked, the pool keeps its first. A malformed RUNNER_POOLS (any field off-schema) is rewritten from runners.json, never crashed on.
Re-dispatch is not here. Jobs already queued on a dead member keep that label; #27114's dead-label janitor re-runs them, and the re-run reads the pool's new member.
docs/ci-runners.md: a "Runner pools" section.

Order of the rest

This merges; its first scheduled watch creates RUNNER_POOLS.
Conversion PRs by workflow group, on main and feat-cmux-next: every literal label in runs-on (fallbacks after || and matrix values included) becomes a pool read. Pickers keep their outputs first; their fallback becomes the pool.
The PR that converts the last reader of LINUX_RUNNER/MACOS_RUNNER_* retires the switch's variable flipping; until then both run side by side.
The lint (tests/test_ci_runner_set.py) lands with the last part: it fails on any label literal in runs-on, with two trust-boundary exceptions matched by exact shape: the fork-owner branch (github.repository_owner != 'manaflow-ai' && 'ubuntu-24.04': forks have no Blacksmith) and the fork pull request literal (a fork PR may not read a variable that could name an owned Mac). The CI_TRUSTED_RUNNER allowlist selectors stay as they are, exempted by the exact shape validate-cla-policy.rb checks, and so do signing jobs (a signing secret plus a literal ephemeral label, the rule-7 classifier), which move through the signing picker instead.

With #27114: its dead-label re-dispatch re-runs jobs queued on a dead label. Once jobs read RUNNER_POOLS, the re-run lands on the pool's next member, and the pool members are the declared set it routes within.

Overlap: #27022 and #27121 change the watch job; this PR adds one step to it and no longer touches cloud_overflow_switch.py. Until #27121 lands, CI_CLOUD_LABEL_HEALTH is unset and only the janitor snapshot is evidence.

Risk

Low. Nothing reads RUNNER_POOLS yet. The new step writes one repository variable with an existing token; on any failure it logs and the switch's outcome is unchanged (the step runs after it).

Tests

tests/test_ci_runner_pools.py (new, ci guard group): declared pools are valid and name no reused machine; evidence is read from a recorded real janitor snapshot (22:53Z 2026-10-28) and the #27121 health shape; a dead member flips its pools; marks expire or clear on a start; every malformed RUNNER_POOLS shape ({"_dead":[]}, naive times, non-members, non-objects) is repaired; one write per change; it runs in watch with the switch token.
tests/test_ci_cloud_overflow_switch.py, tests/test_ci_runner_capability_resolver.py, tests/test_ci_no_github_hosted_macos.py, tests/test_ci_fork_runner_routing.py, tests/test_ci_self_hosted_guard.sh, tests/test_ci_workflow_guards_are_wired.py pass; actionlint is clean on the two changed workflows.

Adoption verification

Updated onto main at 181a0680b20ea39bb930ba5295535ac214569519 using the repository merge helper. This includes the shared rescue-table repair #27485. Exact head: 106dfe877b6609be5a144a6955f3a1f332c704fa.

Observed / Not run

python3 tests/test_ci_runner_pools.py (27), python3 tests/test_ci_owned_pool_sources.py (9), python3 tests/test_ci_cloud_overflow_switch.py (53), and python3 tests/test_runner_label_policy.py (33) pass after the base update. The repository-variable defaults test and actionlint on all five changed workflows passed on the same routing-source diff before that update. git diff --check origin/main...HEAD passes. Future-dated probe evidence is rejected with the same bounded clock-skew rule as snapshots: the focused test fails before the fix (test commit 3bfbf2c3d49) and passes after (a9d973c62e2). Live RUNNER_POOLS exists and validates against the declaration. Independent Astra review: SHIP at 106dfe877b6609be5a144a6955f3a1f332c704fa, no P1/P2 findings after reviewing the full pool diff, future-health repair, and current-base interactions. The exact-head required-check wait passed. No live repository variables were changed, and no nightly, publishing or cancellation was dispatched. Native builds are not needed for this Python/workflow change.

Changelog

none

Agent session: agent session