PR #32514 - manaflow-ai/cmux

What

build-nightly-app on nightly-next: both dedicated build slots on each of the nightly Macs now build from one fixed root per Mac instead of each slot's own workspace, and the nightly-next compile cache limit goes from 3 GiB to 10 GiB when its volume has room.

All logic is in scripts; nightly.yml is unchanged (still under its size cap).

Why (measured on 20 nightly-next runs, 41455121554 to 41485637444)

The cache keys differed per slot. Xcode's compile cache keys hold the absolute source, DerivedData and package-checkout paths. Each slot built in its own workspace into its own kept DerivedData and SourcePackages, so the same module (Atomics) had 4 keys, one per slot. Prefix mapping on Xcode 26.6 doesn't remove that.
The slots pushed each other's entries out of a small cache. All slots on a Mac share one CAS, capped at COMPILATION_CACHE_LIMIT_SIZE=3 GiB. The CAS starts a new generation whenever a build ends above half the limit (1.5 GiB), and keeps only one generation behind it. One build is about 3.3 GB, so every build rotates, and two builds by other slots (other keys) empty a slot's entries. Each slot builds only about every 4th tip, so its entries were nearly always gone. Every run compiled all ~138 Swift module tasks (SwiftCompile (138 tasks) | 1600 s CPU in run 41485637444). The cache replayed only about 311 tasks, the same set every run.
"clean" was almost always chosen. 18 of 19 runs chose mode=clean, because Package.swift or an import or #if line changed since that slot's last build. That build was 3-4 promotions back, so Package.swift (FFI re-pins, new targets: 12 changes this morning) had nearly always changed.

Change

scripts/ci/nightly-steps/app-host-build-root.sh (sourced by app-build-release.sh): on a slot that keeps build state (the dedicated nightly-next builder), it takes flock on the root's lock and never waits for it (scripts/cmux-next/host_build_lock.py, as the DEV and PR release compiles already do). Then:
- It clones the checkout to ROOT/src (canonical-build-root.sh). Before that, it fetches the root's last-built commit into the checkout, so the clean rules can diff against it even when the other slot built it.
- It builds into ROOT/DerivedData, with packages in ROOT/SourcePackages (seeded once from the slot's own) and XCBUILD_FILE_SYSTEM_MODE=checksum-only.
- The root path is the same on every nightly Mac, so keys also match across Macs.
nightly_build_state.py prepare-host / finish-host: the same clean rules as before (no rule relaxed), judged against the Mac's last build. On success, while the lock is held, it marks the build done and clones the products into the workspace's build-universal. Later steps change the app bundle there, never in the root. The slot's own kept DerivedData is dropped, so there is one DerivedData per Mac instead of two.
Fallback: a held lock, a lock error or a failed setup builds on the slot's own state exactly as before, at once. A cold-measurement run (cold_cache) never takes the root.
Compile cache limit: 10 GiB for nightly-next when the CAS volume has at least 60 GiB free; otherwise 3 GiB, as before. Other tracks keep 3 GiB. 10 GiB means a generation rotates at 5 GiB, so about three builds' entries survive. That stays under glaeda's hourly prune, which empties a local CAS past 20 GiB. app-bound-compilation-cache.sh bounds a kept (linked) cache at 12 GiB instead of 5, so a full one isn't deleted.

Is a full clean needed on a Package.swift or import change?

Yes; I kept that rule. Those are the changes behind the audited stale autolink and force-load objects. A clean build is only slow when the compile cache misses. An empty DerivedData plus a warm cache whose keys match replays every unchanged task: 102 s with 4,192 of 4,192 hits, against 600 s at another path. So the fix is matching keys and a cache big enough to keep them, not relaxing the clean rules. Per Mac (every 2nd tip instead of every 4th), the diff window behind each build is also about half as long, so more builds stay incremental.

Shared compile cache: 0 hits (finding, not fixed here)

The nightly reads the fleet cache node (COMPILATION_CACHE_REMOTE_SERVICE_PATH is set; the build uploads outputs). It can't hit, because the only trusted writer fills a different build. The writer's catch-up fill compiles the feat-cmux-next tip's Release compile from the next-release root, with no nightly defines. The nightly builds:

from other paths (this PR moves it to a fixed nightly-next root, the same on both nightly Macs);
with OTHER_SWIFT_FLAGS=$(inherited) -DCMUX_NO_PASSWORD_IMPORT -DCMUX_NO_BROWSER_DATA_IMPORT -DCMUX_BLOCKING_CHECKS, which apply to every Swift target, packages included;
with AppIcon-Nightly and an emptied browser-sources.json.

Every Swift key differs from the writer's. Getting hits takes a writer fill of the nightly-next recipe at this PR's root path; that is a follow-up on the fleet side. Separately, other jobs that share a Mac's CAS with a smaller limit can still rotate it early.

Tests

tests/test_nightly_build_state.py: new HostBuildRoot cases (two slot checkouts, real scripts, stubbed Xcode):
- both slots share one root, and slot 2 builds incrementally on slot 1's DerivedData although its own checkout never held slot 1's commit;
- a structure change since the Mac's last build is clean;
- an unfinished build makes the next one clean;
- a held lock falls back to the slot's own tree;
- the root is off unless the slot keeps build state.
30/30 pass.
tests/test_ci_fleet_compilation_cache.py 9/9 and tests/test_ci_fleet_compilation_cache.sh pass.
tests/test_workflow_file_size.py passes.
shellcheck is clean.
tests/test_nightly_universal_build.sh fails the same way on base ("the short nightly decide gate …").

Performance

Before: the 20 runs above, Build step (Build nightly app (Release)) only. After: to be filled in from the first nightly-next runs once this is on nightly-next. Branch dispatches don't reach the nightly Macs, because placement there requires refs/heads/nightly-next.

Metric	Before	After	Delta	Harness or run link
Build step s, nightly Mac A (median of 9)	587 (383 to 634)	pending		41485637444, 41457449250
Build step s, nightly Mac B (median of 10)	583 (304 to 1304)	pending		41480007529, 41471126151
mode, Mac A	9 clean / 0 incremental	pending		same runs
mode, Mac B	9 clean / 1 incremental	pending		same runs
cache replay hits / local misses per build (both Macs)	about 311 / 45-165 (all ~138 Swift module compiles run)	pending		job logs, `note: replay cache hit` / `local cache miss`
Blacksmith minutes per day	0	0	0	no routing change

Warm sources: the nightly Mac's kept DerivedData and package checkouts (now one per Mac), and the local compile cache on the Mac (10 GiB). The fleet store is read but can't hit yet (above).

Agent session: agent session

🤖 Generated with Claude Code

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

Nightly-next builds no longer recompile all ~138 Swift module tasks every run: both dedicated build slots on each Mac now build from one fixed root per Mac with a 10 GiB compile cache, so Xcode's cache keys match across slots and survive between runs.

Details

Each slot used to build in its own workspace with its own kept DerivedData and packages; cache keys embed those absolute paths, so one module had a different key per slot, the 3 GiB cache rotated on every build (one build is about 3.3 GB), and each slot compared against its own last build several tips back — 18 of 19 runs were clean builds.
The Release build step now locks a per-Mac root with a flock it never waits for, clones the checkout into ROOT/src, and builds into shared ROOT/DerivedData with packages in ROOT/SourcePackages, applying the same clean rules against the Mac's last build. The root path is identical on both nightly Macs, so keys match across them too.
A held lock, lock error, or failed setup falls back to the slot's own state as before; cold-measurement runs never take the root. The cache limit is 10 GiB when the CAS volume has at least 60 GiB free, else 3 GiB, and a kept linked cache is bounded at 12 GiB instead of 5.
The fleet compile cache still can't hit: the only trusted writer fills a different build recipe, so hits from the shared store are a fleet-side follow-up, not fixed here.