ci(nightly): take the cmux-tui download and the light-job placement wait off build-nightly-app's tail

Merged
TE
4d ago
+220-35

build-nightly-app takes about 17 min, and the path from landing to the NEXT feed needs to fit in 30 min. After the Release compile, the job ends with two waits on the feed's critical path. This PR moves both off it. The compile itself is unchanged.

What changes

  • cmux-tui client prefetch. "Bundle the cmux-tui client" downloaded the resolved client and its companions after the compile. Over 16 nightly-next runs on 2026-11-02 the step took 40-121 s (median 50 s), and the download was 31-108 s of that. "Fetch the resolved cmux-tui tree companions" now starts install-cmux-tui-client.sh in the background (background_build.py) on a scratch bundle while the app compiles. It writes into the same CMUX_TUI_CLIENT_CACHE and uses the same manifest URL, expected commit and capability checks. The bundle step joins it and then runs its own install, which reuses a cached slice only if its sha256 matches the attested manifest. A failed or unfinished prefetch is stopped and its cache is deleted, so the install then downloads everything as it does today. The prefetch runs only when the bundle step would also run and the tree is already published. Its token goes into the prefetch command only, not into the pin-cmux-tui.sh fetch in the same step.
  • Light-job placement in the background. "Place the light jobs on a free mini" waits up to 180 s for idle nightly-only runners and was the job's last step: 3-188 s, mean 66 s (70 s in run 41485637444, 188 s in run 41482932276). It now starts before "Archive unsigned nightly app" (light-placement.sh start), so its wait runs during the archive, the uploads and the browser-data proof (about 100-230 s). "Collect the light-job placement" (id: place-light, still the last step before the always() cleanup) joins it and copies its outputs. The light jobs read steps.place-light exactly as before. If the placement fails, the light jobs take Blacksmith as before, and "Stop the helper builds still running" ends the placement if the job fails first.
  • One comment block moves from nightly.yml into browser-data-proof-start.sh to keep nightly.yml under its 490,000-byte cap.

Performance

The before numbers come from run logs. The after numbers are estimates, not measurements.

metricbeforeafter (estimate)delta (estimate)run link
Bundle the cmux-tui clientmedian 50 s (40-121 s, 16 runs)about 15 s (attestation check, install from cache, strip, capability checks)about -35 s median, up to -100 s41485637444 (64 s), 41474606044 (105 s)
Place the light jobs on a free mini (on path)mean 66 s (3-188 s, 16 runs)about 1 s (join; the wait overlaps 100-230 s of archive, uploads and proof)about -65 s mean, up to -187 s41485637444 (70 s), 41482932276 (188 s)
build-nightly-app jobmedian 1118 s (857-2207 s, 16 runs)about 1020 sabout -100 s41485637444 (1013 s)

Blacksmith minutes per day: unchanged. The light jobs use the same placement logic, and the placement now observes capacity about 2-3 min earlier.

Warm sources: none change. Both cuts remove waits after the compile. "Build nightly app (Release)" took 383-1304 s over the same runs, and its kept DerivedData and compilation cache work as before.

Tests

  • tests/test_nightly_trusted_routing.py: new tests run light-placement.sh start and finish against a stand-in placement. They check that the outputs and summary reach the collecting step and that a failed placement fails it. They also check the step order: the start step comes before the archive and the collect step is last before cleanup. A structural test checks that the prefetch uses the bundle step's cache, commit, manifest and capability checks, that it runs only where the bundle step runs, and that the join comes before the install and drops the cache on failure. 47 tests pass.
  • Pass: test_ci_portable_job_placement.py (updated to the start step's id), test_nightly_step_coverage.py (new step registered), test_workflow_file_size.py (489,915 bytes), test_nightly_background_steps.py, test_ci_nightly_off_blacksmith.py, test_nightly_build_state.py, test_ci_resolve_cmux_tui_client_commit.sh, test_ci_self_hosted_guard.sh, test_nightly_delivery_concurrency.py, test_nightly_signer_jobs.py.
  • test_nightly_universal_build.sh fails on the current base with the same message ("the short nightly decide gate must use the pooled Linux route with a Blacksmith fallback"), so that failure is not from this PR.

No started nightly run is cancelled by this change or while landing it.

Agent session: agent session

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Takes two waits off build-nightly-app's critical path so the path from landing to the NEXT feed fits its 30-minute window. The Release compile is unchanged; the estimated job median drops from about 1118 s to about 1020 s.

Cmux-tui client prefetch

  • "Fetch the resolved cmux-tui tree companions" now starts install-cmux-tui-client.sh in the background on a scratch bundle while the app compiles, using the same cache, manifest URL, expected commit, and capability checks as the install.
  • "Bundle the cmux-tui client" joins the prefetch and installs from cache, reusing slices only when sha256 matches the attested manifest; a failed or unfinished prefetch is stopped and the install downloads as before.

Light-job placement

  • The placement now starts in the background before archiving and is collected as the last step, so its wait for an idle mini overlaps the archive, uploads, and browser-data proof.
  • The light jobs read the collecting step exactly as before, and a failed placement still falls back to Blacksmith.