PR #29436 - manaflow-ai/cmux

Leo, 2026-10-31: "i mean the linux things cant our minis just run them" and "we just need the nightlies bro". Rulebook: nightly jobs come first on the minis; Blacksmith is capped, paid overflow; no nightly job is pinned to Blacksmith.

Follows #29130 and #29343 (portable placement: fleet Linux, then a free mini, then Blacksmith), which left the nightly out, and #29353 (dep-fleet-3), which found that every nightly Linux hop waits on blacksmith-32vcpu-ubuntu-2404 because no fleet Linux Actions runner exists.

What moves, and why each needs no Linux kernel

job	what it does	why a Mac runs it the same
`build-nightly-remote-daemon`	cross-compiles `cmuxd-remote` for darwin/linux × arm64/amd64, writes checksums, manifest, notices; uploads the artifact	pure Go with `CGO_ENABLED=0 -trimpath -buildvcs=false` and the pinned toolchain from `go.mod` (the script refuses any other), so the binaries are the same bytes on any host. It used to run inside `build-nightly-app` on a Mac. Hashes use `shasum -a 256`; the Python helpers parse as 3.9. No environment, no secret, `contents: read`.
`await-nightly-tree-marker`	waits for the build commit's tree-commit marker on files.cmux.com	bash, `curl`, `date +%s`, `python3 -c json`. No environment, no secret, `contents: read`.

Both stay on their Blacksmith runner-set member unless a mini is idle the moment they queue.

How

build-nightly-app (already on an owned Mac) gets two last steps before "Stop the helper builds still running", on nightly-next only, not on an unsigned resume:
- mint the routing App token, with runner-list read only (GLAEDA_ROUTE_APP_KEY, a repository secret; cmux-next-guards.yml's portable-placement already mints it this way on pull requests, outside the environment);
- scripts/ci/portable_job_placement.py with JOBS="build-nightly-remote-daemon await-nightly-tree-marker", NIGHTLY=1, FLEET_LINUX="", SIDE_LABEL=vars.CI_SIDE_LANE_RUNNER.
Placement runs right before the two jobs queue, so it reads which runners are idle at that moment. It places a job only on a runner idle now, after subtracting the label's own queue. Anything else (token, runner list or snapshot unreadable, nothing idle, attempt 2+ by a person) sends both to Blacksmith at once. So nothing waits on a busy mini: this is the stall-aware fallback, decided up front.
NIGHTLY=1 is the one change to the script. A nightly run's own jobs skip the "a nightly job waits on this family, so take no mini" refusal, because they are those nightly jobs. Once one is queued on a side label, the janitor counts it as reserved_queued (nightly.yml matches RESERVED_POOL_WORKFLOW), so PR placements stay off that mini while it waits. That is how nightly jobs get admitted ahead of PR jobs.
The two runs-on read needs.build-nightly-app.outputs.light_mini_runner only when light_attempt equals this run attempt, which is set only if the placement step succeeded. A re-run of failed jobs keeps the old outputs and so goes to Blacksmith.
There is no fleet Linux step. A nightly job never takes vars.LINUX_RUNNER as it is, and registering GCP runners for the nightly would change the trust list.
No new trust class: the nightly's Mac jobs already run on owned side and nightly minis. No signing job, environment, secret or permission of any signing or publishing job changes. No cancel-in-progress anywhere.
To stay under nightly.yml's size margin (test_workflow_file_size: 490,000 bytes; base 489,370), two comments on the Chromium engine fetch step are condensed (comments only). The file is 489,912 bytes. (A first try moved "Resolve immutable SSH daemon metadata" to a script; review at d731b470 blocked it because build-nightly-app checks out the resolved build commit, which may predate a new script. Reverted.)

Performance

metric	before	after	delta	harness or run link
queue for `build-nightly-remote-daemon` (on the critical path: app build → remote daemon → prepare → sign → publish)	3.2 min / 3.8 min	about 0 when a side mini is idle (placement takes only idle runners); unchanged otherwise	expected −3.2 to −3.8 min per run	runs 41187578702/2, 41191580829/2, jobs API `started_at − created_at`; after: pending (first nightly-next run after merge)
queue for `await-nightly-tree-marker` (beside prepare/sign, only when the tree is pending)	3.2 min	about 0 on an idle mini	off the critical path	run 41187578702/2
whole-run queue-only time of the published NEXT build	28.5 min of 83.9 min	pending	expected about −3.5 min	NOW stage timing (HQ#5634, HQ#5639), run 41191580829 attempt 2
Blacksmith billed per nightly-next run for these jobs	up to 2 jobs × 1 min minimum × 32 vCPU = 64 vCPU-min	0 when a mini is idle	up to −64 vCPU-min per run	jobs ran 0.1-0.2 min each; Blacksmith bills at least 1 min
Blacksmith per day	up to 64 vCPU-min × runs (about 96 a day at `NIGHTLY_NEXT_MIN_INTERVAL_HOURS` 0.25) ≈ 6,100 vCPU-min	scales with mini idle rate	up to −6,100 vCPU-min/day	as above

The "after" numbers will come from the first nightly-next runs after merge, read from the jobs' runner_name and queue times and from NOW's stage timing.

Warm sources: none needed. Both jobs compile nothing warm-cacheable (warm-sources.json already exempts the daemon build). A mini's Go toolchain stays in its tool cache between runs.

Not in this PR (follow-ups)

prepare-nightly-variant-primary and the prepare-nightly-variant no-op leg are on the critical path too (2.7-4.8 min of queue). The prepared tarball is the signing input, and a Mac's bsdtar can add AppleDouble/xattr entries. Moving it needs a byte-identity check first.
resolve-nightly-cmux-tui-client and pick-nightly-build-runner come before the app build (2.5-4.4 min of queue). Placing them would need placement in decide, which holds id-token: write.
Fleet Linux Actions runners for the nightly (trust list; HQ#5561).

Tests

tests/test_nightly_trusted_routing.py adds a contains evaluator and a new LightJobsOnAFreeMini class. A placed job takes the mini, and only the job that was placed. A missing, stale or re-run placement, or a fork, keeps the old route. Both jobs hold no environment, secret or id-token. Only those two jobs read the outputs. The mint step reads only the routing key, with read permissions only.
tests/test_ci_portable_job_placement.py: with NIGHTLY=1, waiting nightly jobs do not hold the placement off, the label queue still counts, and busy runners take nothing. Also tests the nightly.yml wiring order (mini before Blacksmith, attempt guard).
.github/runners.json classes both jobs portable-checks, and scripts/ci/runner_routing_table.py's portable_order() accepts nightly.yml's order: build-nightly-app's light_mini_runner, then a Blacksmith Linux label, with no fleet Linux step.
tests/test_nightly_sign_runner.py: build-nightly-app joins the jobs allowed to read the routing App key.
.github/nightly-step-coverage.json covers the two new steps.
Run locally: every tests/test_*.py that reads nightly.yml, runners.json, the step coverage or the placement script. The ones that fail locally fail with the same test names on base (local environment). actionlint shows no new findings.

Risk

Medium-low, CI wiring only. If a mini's runner hook refuses the job, it fails at "Set up runner", and the owned-pool rescue's re-run (attempt 2) takes Blacksmith. A build commit older than this PR has the old placement script, so it skips the NIGHTLY override and refuses more often, which keeps the old Blacksmith route. No timer, observer or loop is added.

Changelog

none

🤖 Generated with Claude Code

Agent session: agent session