PR #29353 - manaflow-ai/cmux

Leo, 2026-10-31: "DON'T WE HAVE THE DEDICATED NIGHTLY BUILDERS...". Rulebook: the NEXT nightly runs end to end on our own fleet, and Blacksmith is a capped, paid overflow.

Why the nightly's Linux jobs queue

NEXT nightly run 41187578702 attempt 2 took 101.7 min, 47.2 min of it queued. On the Linux side, every control job between the app build and publication ran on blacksmith-32vcpu-ubuntu-2404 and waited 2 to 6 min for it, even when the job itself ran for 0 s. Three causes, read from nightly.yml on feat-cmux-next and the repository variables:

There is no fleet Linux runner for Actions. The GCP pool serves agent lanes and cmux-ci, not GitHub Actions jobs: no Actions runner outside the minis is registered, and LINUX_RUNNER is blacksmith-32vcpu-ubuntu-2404. Every Linux nightly job uses the runner-set expression from #27125 (vars.LINUX_RUNNER == 16 vCPU && 16 vCPU || 32 vCPU), so all of them resolve to Blacksmith 32 vCPU, the label with the longest queue (409 queued, oldest 45 min at 12:08Z).
The trusted set for nightly Linux jobs is {Blacksmith 16, Blacksmith 32} (tests/test_nightly_trusted_routing.py, #28209). A free owned mini is not in it, and #29130 (portable placement: fleet Linux, then a free mini, then Blacksmith) explicitly leaves the nightly out.
Serial hops. The Linux critical path is remote daemon, then prepare primary, then the prepare fallback, then the prepare-nightly-variant aggregate, then signer-app, then signer-dmg, then publish-nightly. Each hop pays a fresh Blacksmith queue.

This PR fixes the part that needs no new runner or trust change: one of those hops. Routing these jobs to owned capacity is a follow-up (see "Not in this PR").

What changes

prepare-nightly-variant was a job with one echo step. It succeeded when the primary or the fallback preparation did, so later jobs could keep a single need. In run 41187578702 attempt 2 it queued 4.8 min and ran 0 s, on the signing path.

The fallback job now carries that result itself. It is renamed from prepare-nightly-variant-fallback to prepare-nightly-variant, and the echo job is deleted. Nothing else about the job changes: its needs, its if (it never skips after a primary success or failure), its runner choice (the primary's label after a success, the other set size for a retry), and its steps. Its result is the same as the old aggregate's:

primary succeeded: the no-op leg succeeds, so the result is success;
primary failed: the result is the fallback preparation's result;
primary skipped or cancelled: the job is skipped, as the aggregate was.

None of the jobs that read needs.prepare-nightly-variant change: signer-app, pick-nightly-sign-runner, build-sign-notarize-nightly, legacy-identity-target and report-nightly-failure. PREPARE_FALLBACK in the primary's env compares github.job with the new id. That is still false for prepare-nightly-variant-primary, and the fallback still sets it to 'true' itself.

The tests that named the fallback job by id now use the new id. The aggregate's entries in .github/nightly-step-coverage.json and tests/test_ci_macos_xcode_selection.py are removed. The fallback's matrix legs are now named prepare-nightly-variant (<variant>). That is the name scripts/ci/nightly_sign_admit.py (the nightly-sign.yml shadow, off unless vars.NIGHTLY_SIGN is shadow) selects. After a primary success the no-op leg records no digest, so admission now takes the digest of the successful prepare-nightly-variant-primary (<variant>) leg, whose upload the no-op leg reused, and names that job as the producer. A leg with no record and no successful primary leg is still refused. New tests in tests/test_nightly_sign_admit.py cover both cases.

Performance

metric	before	after	delta	harness or run link
Linux hops on the nightly-next signing path (app build → publish)	7	6	−1	`needs:` graph of nightly.yml
queue on the removed hop (`prepare-nightly-variant`)	4.8 min	0	−4.8 min	run 41187578702 attempt 2, REST jobs list (started_at − created_at)
Linux queue on the critical path (remote daemon 3.2, prepare primary 2.8, fallback 4.8, aggregate 4.8, signer-app 5.6, signer-dmg 2.2, publish-nightly 4.7)	28.1 min	0.4 min (26 s: remote daemon 1 s, prepare primary 1 s, prepare-nightly-variant 1 s, signer-app 2 s, signer-dmg 18 s, publish-nightly 3 s)	−27.7 min	before: run 41187578702 attempt 2; after: run 41217892491 (published build 4121789249101), REST jobs list (started_at − created_at)
whole-run queue	47.2 min	1.3 min (76 s over 45 jobs: Linux 64 s, Mac 12 s)	−45.9 min	before: NOW (HQ#5634) 12:00Z; after: run 41217892491
whole critical-path queue, Linux + Mac (app build → publish-nightly)	not measured separately	0.6 min (33 s)		run 41217892491
Blacksmith billed per nightly run for the removed job	1 job × 1 min minimum × 32 vCPU = 32 vCPU-min	0	−32 vCPU-min per run	Blacksmith bills each job at least one minute
Blacksmith per day	32 vCPU-min × runs per day (at most 96 a day with `NIGHTLY_NEXT_MIN_INTERVAL_HOURS` 0.25)	0	up to −3,072 vCPU-min/day	as above

The "after" queue numbers come from run 41217892491, the first published nightly-next run after this merged. Most of the drop comes from less contention on the Blacksmith 32 vCPU label at that hour and from the nine nightly slots (cmuxterm-hq#5641), not from this PR alone; this PR's own share is the removed hop.

Not in this PR (follow-ups)

Owned capacity for the nightly's light Linux jobs. The secret-free ones on the path (build-nightly-remote-daemon, await-nightly-tree-marker, the prepare no-op leg) could take an idle owned Mac, picked in the step before them, so no picker hop is added. That widens the trusted set in test_nightly_trusted_routing.py. The nightly app build already runs on owned side and nightly Macs, so this adds no new trust class. It builds on #29130's placement once that lands.
Fleet Linux Actions runners. Registering GCP pool VMs as Actions runners for the nightly changes the runner trust list, so it is not in this PR.
The release-environment jobs (signer-app, signer-app-notary, signer-dmg, publish-nightly and the other publish jobs) keep their Blacksmith labels. The trust and keychain rules for them are unchanged.

Risk

Low. This is CI wiring only, with no change to environments, secrets, permissions, runner labels or any step. The one behaviour change: if the primary preparation succeeds but the fallback's no-op leg itself fails (for example at "Set up runner"), signing no longer runs. Before, the aggregate job would have run anyway, though it needed a runner of its own too. No timer, observer or loop is added.

Observed / Not run

Local reads: the YAML loads, no prepare-nightly-variant-fallback reference is left, and the needs graph is acyclic.
The nightly guard tests that read nightly.yml run in this PR's guard group. Their results are in the checks.
Not run: a nightly on this head. The first nightly-next run after merge is the measurement.

Changelog

none

🤖 Generated with Claude Code

Agent session: agent session