PR #33348 - manaflow-ai/cmux

Leo:

The last build took 37.5 minutes end to end: 18.8 minutes compiling and 15 minutes signing and publishing.

is it as warmed/sharded/skipped of redundancies as possible tho? and also the signing/publishing don't require 30 cores

This PR is the warm half. The signing/publishing cores are another PR.

Leo-ask: ask-217810e1b3

What was wrong

build-nightly-app takes the warm path (kept DerivedData in the Mac's host build root, no R2 restores) only when
nightly_build_state.py prepare nightly-next writes a mode. For a *-glaeda-nightly-only-N runner it did so
only with NIGHTLY_ONLY_BUILDER=true, which holds only when pick-nightly-build-runner read the runners and
offered glaeda-nightly-only-m4pro-builder. That read (GET /actions/runners with the routing App) got HTTP 403
(rate limit) in most recent runs, so the picker fell back to the plain glaeda-nightly-only-m4pro label, mode
stayed empty, and the job restored the compilation cache and the Swift packages from R2 and compiled colder.
Four of the last six nightly-next runs went that way.

run	runner	mode	Restore Xcode compilation cache	Cache Swift packages	Build nightly app (Release)	job
41570134320	cmux7s-mac-mini-glaeda-nightly-only-1 (builder label)	incremental	skipped	skipped	562 s	1071 s
41586556429	cmux7s-mac-mini-glaeda-nightly-only-5 (plain label)	none	83 s	185 s	413 s	1064 s

Across the six runs, cold builds took 413-1567 s in the build step, plus about 270-320 s of restores.

The fix

dedicated() now covers every nightly-only runner, without the builder flag, and NIGHTLY_ONLY_BUILDER is
gone from the workflow and from nightly_build_state.py. The picker still offers the builder label when it
can read the runners; it only stops deciding whether the build is warm.

Why concurrent slots on one mini are safe:

The Release build takes the Mac's one host build root (app-host-build-root.sh,
/Users/Shared/cmux-build-fleet/xcode/nightly-next) under a flock that is never waited for
(host_build_lock.py); fd 9 is inherited by xcodebuild, so the lock lives as long as the build does.
A slot that finds the lock held builds in its own kept tree,
~/Library/Caches/cmux-nightly-next/<runner>/<sha256(workspace, toolchain)>. That tree is keyed by the runner
name, so no two slots share one, and its own incremental.json checkpoint keeps the same clean rules.
The slot's next host-root build deletes it (finish_host), so a mini does not keep one DerivedData per slot.
mark-built only marks a checkpoint under the slot's own runner directory, and only when prepare linked
build-universal there.
The workflow's mode conditions are unchanged: with a mode, the R2 restores, the package sanitize and the
cache save are skipped, and mark-built runs (tests/test_nightly_trusted_routing.py passes unchanged).

Disk: each nightly-only slot now also keeps its notices crate cache (bounded at 8 GiB by notices_cache), and
a fallback build leaves a slot's DerivedData until that slot's next host-root build.

Design surface: CI only, no UI.

Performance

metric	before	after	delta	harness or run link
R2 restores in build-nightly-app (compilation cache + Swift packages), plain-label run	268 s (83 + 185, run 41586556429)	0 s (skipped, as in builder-label run 41570134320)	-268 s	job step timings, nightly.yml build-nightly-app
nightly-next runs that built cold because the runner read failed	4 of 6	0 expected	-4	last six nightly-next runs
Build nightly app (Release), cold vs warm	413-1567 s cold	~550 s warm	varies by tip	same runs

The "after" column is the measured builder-label run, not a run of this head: the first nightly after the land
gives the real number.

Observed / Not run

Observed:

tests/test_nightly_build_state.py (30 tests, the two new ones fail on base and pass here),
tests/test_nightly_build_runner.py, tests/test_nightly_trusted_routing.py, tests/test_ci_warm_sources.py,
tests/test_nightly_dedicated_mini.py, tests/test_workflow_file_size.py, tests/test_ci_runs_on_fallback.py
pass locally.
scripts/cmux-next/prepush-gate.py: no new violations.

Not run:

A nightly at this head. The next nightly-next run after the land is the check: its
"Start Release DerivedData empty" log should print a clean or incremental mode on any nightly-only runner.

Agent session: agent session

🤖 Generated with Claude Code

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

Fixes nightly-only nightly-next builds running cold after the runner picker failed to read the runners.

The picker's runner read returned HTTP 403 in four of the last six nightly-next runs, so those jobs took the plain glaeda-nightly-only-m4pro label, restored the compilation cache and Swift packages from R2 (~270–320 s), and compiled from empty DerivedData. Every *-glaeda-nightly-only-K runner now keeps warm state (dedicated() covers all of them), and NIGHTLY_ONLY_BUILDER is removed from the workflow, nightly_build_state.py, and tests.

Concurrent slots on one Mac stay apart:

The Release build takes the Mac's single host build root under a never-waited flock.
A slot that finds the lock held builds in its own kept tree keyed by its runner name; that tree is dropped by the slot's next host-root build.
mark-built only writes a checkpoint under the slot's own runner directory.

CI-only; no migration or rollout steps.