PR #29130 - manaflow-ai/cmux

Leo, 2026-10-31: "i assume blacksmith prob cut us off cuz we were using like $10k a day so", "y'know, we'll have to be more self-reliant and size the stuff better", and "i mean the linux things cant our minis just run them" (rulebook: leo-workspace#1867).

What changes

The jobs below need neither Linux nor Blacksmith. Their runner is now picked in this order:

fleet Linux: vars.LINUX_RUNNER, when it names one of our own Linux runners (not a Blacksmith label or a GitHub-hosted image);
a free owned mini: the side label (vars.CI_SIDE_LANE_RUNNER), or the same label on a wider class. The mini must be idle at placement time, after subtracting the jobs the janitor snapshot already shows queued on it. A mini takes nothing while any nightly or release job is queued on its pool family (reserved_queued), so nightly jobs still come first;
their existing Blacksmith label, with its CI_CLOUD_OVERFLOW_SAVED failover. It stays the last || term.

Job	Workflow	Order
cmux-tui `build aarch64-apple-darwin`, `build x86_64-apple-darwin`	cmux-tui-artifacts.yml → cmux-tui-build-package.yml	free mini first, built natively on the Mac path (the path the Linux build's macOS fallback already uses), then the Linux cross build on Blacksmith 32 vCPU. Both targets or neither.
`cmux-next checks`	cmux-next.yml	fleet Linux → free mini → Blacksmith 16 vCPU
`cmux-next CI guard group`	cmux-next-guards.yml	fleet Linux → free mini → Blacksmith 32 vCPU
`nightly-guards / workflow-guard-tests / release-notary`	cmux-next-guards.yml → ci-guards.yml (new optional input `guard_tests_runner`)	fleet Linux → free mini → `vars.LINUX_RUNNER` / Blacksmith 4 vCPU
`tooling-tests`	cmux-next-guards.yml	fleet Linux → free mini → Blacksmith 4 vCPU (16 vCPU under the overflow switch)

scripts/ci/portable_job_placement.py: the decision. It reuses side_lane_placement.side_route(), pr_runner_pool.live_owned_free() and the janitor snapshot. Anything uncertain places nothing on a mini: runners it can't read, no snapshot or a stale one, a person's re-run, attempt 3 or later, no side label, or the AWS pool. Each job then keeps its Blacksmith route.
A portable-placement job in cmux-next.yml and in cmux-next-guards.yml runs the base branch's copy of the script, the way macos-placement does, using the routing App's runner read with cmux-bot as the fallback. Every step fails open. The placed jobs read its outputs only in the attempt that placed them (placed_attempt), so a re-run of failed jobs never reuses a stale mini.
In cmux-tui-artifacts, tree-preflight makes the same call, all or none, and passes darwin_mac_runner to the build. darwin_builder and the tree key stay linux, so the 32 vCPU cross-compile shadow does not start. A re-run takes macos_retry_runner, as the Mac path always has.
On a mini, a job runs scripts/ci/reset-stale-submodules.sh right after checkout, because a reused workspace keeps the previous job's submodule checkouts. The guard group's --jobs "$(nproc)" becomes getconf _NPROCESSORS_ONLN, since macOS has no nproc.
Routing-table guard (scripts/ci/runner_routing_table.py, .github/runners.json): new class portable-checks ("needs": "any"). A job in that class fails the check unless its runs-on takes portable placement's fleet Linux output, then its mini output, and ends on a Blacksmith Linux label. mini_runner counts as a macOS route, so these jobs are listed.
tests/test_ci_portable_job_placement.py (registered in tests/test-execution.toml, run in the ci guard group) covers the order, the nightly hold, queued jobs taking idle runners first, the std spill, all-or-none, every uncertain case, the output format and the wiring in each workflow. The existing runs-on tests are updated: test_feat_ci_guard_group.py, test_cmux_next_checks_workflow.py (checks and the placement job now count as able to run on a mini and get the submodule reset), and test_ci_linux_single_size.py (counts lowered: ci-guards 6→5, cmux-next-guards 10→9).

Performance

These are Blacksmith vCPU-minutes per day that move off Blacksmith whenever a mini is free (or all of them, once vars.LINUX_RUNNER names fleet Linux). The coordinator measured the "before" column on 2026-10-31. The "after" column is the most that can move. How much actually moves depends on how often minis are idle, and each placement job's summary records it.

metric	before (Blacksmith vCPU-min/day)	after (Blacksmith vCPU-min/day, minis free)	delta	harness or run link
cmux-tui build aarch64-apple-darwin	349k	0 (mini)	−349k	coordinator's 2026-10-31 Blacksmith usage read
cmux-tui build x86_64-apple-darwin	339k	0 (mini)	−339k	same
cmux-next checks	264k	0 (fleet Linux / mini)	−264k	same
cmux-next CI guard group	221k	0 (fleet Linux / mini)	−221k	same
release-notary guards, tooling-tests	not measured (4 vCPU, under 1 min each)	0	small	coordinator's 2026-10-31 Blacksmith usage read
total	≥1,173k	0 at full mini availability	up to −1,173k/day	coordinator's 2026-10-31 Blacksmith usage read

Placement adds one 2 vCPU job of about 15 s per run, on fleet Linux when LINUX_RUNNER names it.

Risk

risk:normal, CI routing only. Two risks:

A check that behaves differently on macOS. The guard group runs head and base on the same machine, so a red that comes from the machine shows up as a base red. checks, tooling-tests and release-notary have no base comparison: a macOS-only red there would turn a PR red only when it lands on a mini. The probe run (below) runs them on a mini before this lands.
Two placements racing for the same idle runner. The loser queues on the side label behind one job. The idle count already subtracts the jobs the snapshot shows queued there.

No timer, observer or loop added: placement is one bounded read (60 s deadline) per run.

Not touched: nightly workflows (none of these jobs is nightly; test_ci_nightly_off_blacksmith.py passes), signing, and cmux#29033 (the react-apps shards).

Observed / Not run

Passed locally: tests/test_ci_portable_job_placement.py (18), test_feat_ci_guard_group.py, test_cmux_next_checks_workflow.py (165), test_ci_linux_single_size.py, test_ci_fork_runner_routing.py, test_supplychain_security.py, test_ci_side_lane_placement.py, test_ci_nightly_off_blacksmith.py, test_ci_test_execution_registry.py, test_ci_tui_routes_avoid_nightly_label.py, test_ci_reusable_workflow_permissions.py, test_ci_linux_guard_routing.py.
Red on the head with the same failures as base 7fc75a9: test_runner_routing_table.py (stable-next.yml and nightly.yml entries), test_ci_blacksmith_linux_failover.py (app-platform.yml, cmux-next-chromium-currency.yml), test_ci_runs_on_fallback.py (nightly.yml), test_ci_signing_keychain_is_deleted_before_bundle_code.py (stable-next.yml). runner_routing_table.py check reports the same 5 problems as base, and none for the new jobs.
On macOS (run by hand, the way a mini would run them): checks and tooling-tests steps gave 81 pass and 5 fail. Two were macOS differences, fixed here: the web lint pre-push test read nice (BSD nice prints usage), and package_deps_heal.offender() passed \s, \w and \b to git log -G. macOS's regex has none of those, so the heal never found the offending commit there; it now uses POSIX classes. The other three ("No new SwiftUI", "Scrollbars thin", "Package conventions lint") are continue-on-error warnings that report findings in base content.
Not run: the jobs on a real mini, in this PR's own run. That run uses the base branch's placement, which doesn't exist yet, so it routes the old way. A probe commit forced them onto the side label for one run (then reverted). They waited over 50 minutes with no free mini, because every side runner was busy, and the next push cancelled them. So there is no on-mini result yet. That wait is the case this PR avoids: placement sends a job to a mini only when that mini is idle now.

Routing classifiers

On a mini, a job that portable placement places still runs only its own scripts, none of the app's Mac tiers. scripts/ci/cmux_next_route.py (PORTABLE_RUNNERS) and scripts/ci/detect_ci_change_areas.py (the guard_tests_runner term) therefore read these runners as Linux. Without that, an edit to checks would run every Mac tier, and a change to any guard test the release-notary job names would count as reaching the app. That made test_ci_nightly_push_throttle.py and test_cmux_next_route.py red on this branch; both now pass, and test_ci_change_areas.py fails per test exactly as on base. Main's owned-pool rescue must also watch cmux-next-guards.yml (test_ci_owned_pool_sources.py). That is cmux#29343, into main; this PR lands after it.

Fleet Linux

RUNNER_POOLS.linux and LINUX_RUNNER still resolve to Blacksmith 32 vCPU. No self-hosted Linux Actions runner is registered: the fleet dashboard's runner list is all glaeda Mac labels. So no job can take fleet Linux yet. This PR reads vars.LINUX_RUNNER as step 1 whenever it names a non-Blacksmith label. Once a GCP pool runner label exists and the variable names it, these jobs take it with no further change. Registering those runners is a runner-trust change and is not in this PR.

Changelog

none

🤖 Generated with Claude Code

Agent session: agent session