PR #25731 - manaflow-ai/cmux

Leo, 2026-10-26: "And I thought we were incremental. You keep saying, yeah, we're gonna do the incremental building. Nobody has come back with the incremental building results." Earlier: "like how fast can we just have it, like hot and incrementally built?... reusing as much as you can".

What this does (2 of 2, on top of #25729)

The cmux NEXT Nightly build (nightly.yml build-nightly-app, nightly-next push runs) now reuses its compiled, unsigned app when nothing it is built from has changed. The DEV lane already does this (#24963, #24964). This change shares that code instead of copying it.

Key. The key is job_input_fingerprint.py nightly-release-product. It uses the DEV product's closure: the whole tree minus docs, tests and other products. It also keys:
- the whole nightly.yml;
- every decide output the build reads (identity, icon, arches, browser-import flags, password_readers_out);
- Xcode, the SDK, the arch, the bundled cmux-tui tree key and a recipe.
The fingerprint job's name is part of the key, so a nightly key never equals a DEV key.
Records. Reuse runs only for nightly-next push builds that are not fast, build-only or notary tests. A record is taken only from this repository's own nightly.yml push run on nightly-next whose build-nightly-app finished successfully. The archive's SHA-256 is checked against the record. Records and archives use their own names (cmux-nightly-next-release-product-*, cmux-nightly-next-release-archive-*). A DEV record is never read as a nightly product, and the reverse never happens either. Pull-request runs cannot write any of this.
On a hit. The app and its dSYMs are unpacked into build-universal/Build/Products/Release. Every step that only feeds xcodebuild is skipped: web bundles, Rust helpers, Chief, mobile host, CEF fetch, notices, xcodebuild, Iroh drop, Computer Use helper, strip. These steps still run on a hit:
- the password-import proof;
- the SSH daemon assets (per-run build number);
- the CEF report;
- Sentry;
- the unsigned-app archive and dSYM handoff.
The version, build number and CMUXCommit are stamped later, in build-sign-notarize-nightly, so the compiled products name no run. Signing, notarization and publication always run fresh.
Recording. After a successful build, the products are packed and recorded with a 2-day retention. A build missing the Chromium engine or the remote browser host (both optional inputs) is never recorded.
Misses. Every problem counts as a miss with a reason, and the job compiles as before. That includes:
- an API error or the deadline (240 s, no gh wait);
- a digest mismatch;
- an unsafe tar member;
- a build commit whose nightly.yml differs from the running one.
An unpack is staged and renamed into place, so xcodebuild never merges into a partial app. vars.CI_NIGHTLY_NEXT_PRODUCT_REUSE=0 switches the lookup off.

Baseline (before)

Measured with the same harness (measure_next_build.py --workflow nightly.yml --branch nightly-next --event push --limit 10 --steps). The xcodebuild seconds and cache counts come from each job's log.

run	sha	runner	build step (min)	xcodebuild (s)	cache hit/miss	commits since previous	Swift files
40380339203	8b3b18916	cmuxs-mac-mini-5-glaeda-3	13.0	779	364/136	1	0
40378565252	c2276c463	cmux8s-mac-mini-glaeda-2	9.9	596	364/136	6	7
40372260966	1b4a6766a	cmux14-glaeda-2	11.2	672	364/136	3	0
40370780830	3ce8dfda5	cmux12s-mac-mini-glaeda-3	8.1	488	415/85	2	0
40370046046	a9d1434de	cmux10s-mac-mini-glaeda-5	8.7	524	415/85	14	13
40357982361	3660113fa	cmuxs-mac-mini-3-glaeda-5	8.7	524	447/53	11	13
40351269110	61ea34bde	cmux9s-mac-mini-glaeda-5	10.3	621	415/85	428	579
40203748160	c0cb858c4	cmux14-glaeda-5	7.7	461	366/134	16	49
40197613419	8f6a3f372	cmuxs-mac-mini-6-glaeda-5	7.8	468	404/96	30	14
40197090763	09a881fde	cmux13s-mac-mini-glaeda-2	8.6	514	0/500	-	-

Median build step: 8.7 min. The build-nightly-app job takes 15.6-28.2 min. Landing to publish: median 88.8 min.
Every build is WMO Release, arm64, on a different owned runner instance, and all 134 Swift module compiles run every time.

Risk

Medium. The change touches the shipped nightly's build path, but it is all-or-nothing and content-keyed:

On a miss (the default until a record exists), every step runs exactly as before.
On a hit, the shipped bytes are the app that an earlier successful nightly-next build compiled from identical inputs, after the same post-processing.
Main's nightly and RC builds are untouched, because the lookup runs only for track nightly-next.
A wrong hit would ship an app compiled from other inputs. The tests check the key's coverage (each decide flag, the toolchain, nightly.yml, app sources) and that the skipped steps read no decide output or repository variable outside the key.

Signals

Success: a nightly-next run's step summary says nightly-next Release product: **hit** and the build step is skipped. Its later jobs sign, notarize and publish as usual.
Failure: a run whose lookup says hit but whose later checks fail. To switch reuse off, set vars.CI_NIGHTLY_NEXT_PRODUCT_REUSE=0.

Tests

tests/test_nightly_release_product_reuse.py (new) covers:
- the pack/unpack round trip (symlinks, hard links, dSYMs);
- tar safety: .. and absolute paths, symlink chains, case-insensitive aliasing, members under a link, duplicates, oversized PAX headers, member and byte bounds;
- a build missing CEF is never recorded;
- the running-workflow check, and a miss clearing the products directory;
- workflow wiring: every decide flag reaches the key, skipped steps read nothing unkeyed, and only nightly.yml uses the lane.
Key and namespace tests are in the first PR (tests/test_nightly_release_product_key.py).
Updated tests/test_gh_budget_wait.py (the lookup makes no gh call) and .github/nightly-step-coverage.json.

Agent session: agent session

🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need.}