ci(nightly): the nightly-next Mac compile records the Linux build's REF

TE
4d ago
+719-30

HOLD: paused pending Leo's Apple SDK/tools-on-Linux decision

Leo, 2026-10-27, about the nightly compiling on a GitHub-hosted Mac: "WHY / THESE ARE SLOW AS STUFF / WHERE THE F*** IS LINUX".

What

The Linux Darling whole-app build plans from the stored build rules of one real Mac build of the same shape (its REF). #25740 made the DEV compile record its own REF. This PR does the same for the nightly-next Release compile (build-nightly-app, "Build nightly app (Release)").

Which file: nightly-next builds from the nightly.yml on the nightly-next branch, which is a feat-cmux-next commit. main's nightly.yml builds classic cmux NIGHTLY (a different app), so this PR changes feat-cmux-next's copy.

  • Off by default. It runs only when the new repo variable NIGHTLY_HYBRID_REF_CAPTURE is 1, and only on the nightly-next track.
  • Owned minis with cmux-ci only. On attempt 1, build-nightly-app runs on vars.CI_SIDE_LANE_RUNNER (glaeda-side-std-xcode-26.6, the owned side pool: run 40380339203 ran on cmuxs-mac-mini-5-glaeda-3, run 40378565252 on cmux8s-mac-mini-glaeda-2). Rescue attempts and every fallback run on vars.MACOS_RUNNER_26 (blacksmith-6vcpu-macos-26, e.g. run 40389706045). A Blacksmith or GitHub-hosted Mac has no cmux-ci and can't reach the fleet's private store, so it never captures. The decide step requires runner.environment == 'self-hosted', a *-glaeda[-N] runner name and an executable cmux-ci (the job's $HOME first, then the build user's home).
  • Decides like the DEV compile (#25866). The decide step calls hybrid_ref_capture.py have --budget 30 under the nightly-next profile. That counts complete captures of the key (13 days) and incomplete ones (1 day), and waits out a rate limit for at most 30 s: it runs before the nightly compile. A capture is needed when there's no complete one. Then compile=true unless one came out incomplete in the last day. A lookup that still fails compiles and captures anyway and fails the step red, so a capture can't be lost silently. If the job's start time can't be read, the deadline counts from 30 minutes earlier, which only brings it sooner.
  • Never from a reused product. nightly-next now reuses a compiled app when its inputs didn't change (#25731). The decide step runs before "Look up the nightly app compiled on the same inputs". On compile=true it hands that lookup SKIP_REASON "a REF capture needs a real compile", and the capture starts only when the lookup missed.
  • Keyed on the nightly's own build. hybrid_ref_capture.py gets a nightly-next profile (HYBRID_REF_PROFILE). Its key hashes the project inputs that #25740 also hashes (Package.resolved, cmux.xcodeproj, the macOS xcconfigs and the local package manifests), the toolchain, and the nightly's own recipe in place of the DEV one: nightly-swift-defines.sh, the text of the "Build nightly app (Release)" step (its xcodebuild flags are inline), and the env that step reads (BUILD_TRACK, SWIFT_IMPORT_FLAGS, EMPTY_BROWSER_SOURCES, NIGHTLY_COMPILE_ARCHS, CHANNEL_ICON_NAME). The DEV key doesn't change (a test checks this). A capture runs only when no complete, readable capture exists for the key: artifact hybrid-ref-capture-nightly-next-<key>, from the last 13 days.
  • The compile: on a capture run, the only change is EMIT_FRONTEND_COMMAND_LINES=YES plus a named xcodebuild log (CMUX_XCODEBUILD_OUTPUT_LOG). A non-capture run is unchanged.
  • Under the nightly.yml size cap (#25879). The decide, start and finish run blocks live in scripts/ci/nightly-steps/hybrid-ref-{decide,start,finish}.sh, so this adds about 6 KB to nightly.yml (475 KB; the cap is 490 KB). The tests read the scripts back in place through tests/nightly_workflow_text.py.
  • Bounded, so signing barely waits:
    • Right after the compile and the helper bundling (which changes only the app), one clonefile(2) snapshots the workspace. That takes about a second, and build-universal, the DerivedData, is inside it.
    • The capture then packs and uploads in the background at nice 10 (background_build.py), alongside the rest of the job (notices, the CUA helper, daemon assets, the unsigned-app upload, Sentry).
    • At the end, "Finish the Linux build's REF capture" waits at most 4 minutes, and never past minute 130 of the job's 150. A capture that hasn't finished by then is stopped and uploads no manifest; the next build with the same key takes it.
    • The capture also stops itself at minute 130 (capture --deadline) and on SIGTERM from the job's cleanup. Each upload runs in its own process group, which is killed however the upload ends, so nothing an upload started outlives the job.
    • It doesn't start once the job is more than 100 minutes old.
    • The decide step's job-start gh call never waits out a rate limit (GH_BUDGET_WAIT_MAX_SECONDS: "0"), so it spends none of the job's one-hour gh budget; the have lookup waits 30 s at most. That lookup joins the DEV one in tests/test_ci_no_github_token_polling.py's allowlist.
    • Every new step is continue-on-error with a timeout, and the always() cleanup stops a capture that is still running.
  • Storage: packs go only to the build fleet's private store (cmux-ci artifact-put --lane hybrid-ref-nightly-next). The public manifest artifact (14 days) lists only names, sizes and sha256 values.
  • No new permission. The job keeps contents: write, actions: read, attestations: write, id-token: write. Listing artifacts uses the existing actions: read.

Manifest differences from #25740 (for the HQ assembler)

  • The kept xcodebuild log now starts with the wrapper's xcodebuild command: line (run-xcodebuild-with-diagnostics.sh writes it into a caller-kept log), so xcodebuild_args is filled in.
  • A runner that links .spm-cache to its kept SourcePackages gets a real clone of the packages in the snapshot; the capture refuses a linked .spm-cache.
  • Driver modules come from the builtin-SwiftDriver line's -module-name (#25903): the app target isn't its module.
  • New field "profile": "nightly-next" here, "dev" on DEV manifests. This is additive and schema stays 1.
  • settings holds the five nightly names above instead of CMUX_NEXT_*.
  • workspace is the runner's $GITHUB_WORKSPACE, and derived_data is $GITHUB_WORKSPACE/build-universal. ws-src.tgz leaves out build-universal, which ships once, in build-noobj.tgz/xcbuilddata.tgz. Swift packages are cloned to .spm-cache in the workspace (-clonedSourcePackagesDirPath .spm-cache), so their checkouts are in ws-src.tgz; their git mirrors are not.
  • xcodebuild_args starts with env CMUX_SKIP_ZIG_BUILD=1 xcodebuild ... (the nightly's wrapper line), and the build is universal (ARCHS=arm64 x86_64) unless compile_archs is arm64.
  • Artifact names: hybrid-ref-capture-nightly-next-<key> for a complete capture and hybrid-ref-capture-nightly-next-incomplete-<key> for an incomplete one, the same names have reads. DEV keeps hybrid-ref-capture-[incomplete-]<key>.

Release surface

.github/workflows/nightly.yml and .github/workflows/cmux-next.yml are both on .github/release-surface.txt. So the nightly-next promotion whose range contains this commit waits for an approval in release-surface-ack, as #25740's range (cmux-next.yml) already does. hybrid_ref_capture.py isn't added to the surface: build-nightly-app holds no signing, notarization or publishing secret, and the other scripts/cmux-next/ files it runs aren't listed either.

Risk

Low for the shipped app. With the variable unset, no new step runs, and the compile step only gains two empty env values. On a capture run the shipped compile gains EMIT_FRONTEND_COMMAND_LINES=YES, which only prints the frontend lines. That makes the Actions log larger, the same as on #25740's DEV capture runs. The worst case adds about 10 minutes to the job: decide 1, start 2, finish 5, manifest upload 2. The typical case adds 0 to 4 minutes, once per REF key.

Tests

  • tests/test_cmux_next_hybrid_ref_capture.py (34 tests):
    • the nightly key moves with the nightly step, its settings and defines, and not with the DEV recipe or another step's comment; the DEV key and DEV artifact names are unchanged;
    • have reads the nightly profile's own names, and the upload step writes the same ones;
    • a failed lookup compiles, captures and fails red; a capture run makes the product lookup miss;
    • a DerivedData inside the workspace is packed once, and a linked .spm-cache is refused;
    • the deadline, a failed upload and the cleanup's SIGTERM each leave no uploader or helper running (the test is its own subreaper and checks the helper died by SIGKILL);
    • the real xcodebuild wrapper writes the command line into the kept log;
    • the steps are off by default, owned-mini only, continue-on-error and bounded, and add no permission.
  • tests/test_ci_no_github_token_polling.py: allowlist entry for the nightly have lookup.
  • tests/test_gh_budget_wait.py: a step that never waits (GH_BUDGET_WAIT_MAX_SECONDS 0) no longer needs the 65-minute step timeout.
  • .github/nightly-step-coverage.json: entries for the three new steps.
  • Run locally on the merged tree: the capture suite (also 180 runs of its three cleanup tests, no failure), test_workflow_file_size.py, the polling, gh-budget, helper-ordering, product-reuse and xcodebuild-wrapper tests all pass. Nine other tests that read nightly.yml fail here exactly as on a clean base checkout. actionlint reports the same findings as base.

Stop condition: merged, then NIGHTLY_HYBRID_REF_CAPTURE=1, and the first nightly-next capture's manifest artifact exists.

Agent session: agent session

🤖 Generated with Claude Code