PR #32378 - manaflow-ai/cmux

Coordinator's order: "run the launch smoke in its own Mac job while the DMG notarizes, but publish still waits on its result, so the fresh-home smoke keeps gating."

What changes

On the release-pool path (nightly-next), the fresh-home launch smoke used to run in verify-final-dmg-nightly, after signer-dmg had signed, notarized and stapled the DMG. Now:

launch-smoke-nightly (new, secret-free Mac job) needs signer-dmg-image only. It downloads the unsigned DMG signer-dmg-image uploaded, checks it has the digest signer-dmg checks before signing (UNSIGNED_SHA256), records its image digest, and runs the same launch-smoke-gate.sh on it. It runs while signer-dmg signs, notarizes and staples.
verify-final-dmg-nightly keeps every final-DMG check: the byte digest and the stapler/Gatekeeper/license checks (verify-final-dmg.sh), Apple's checks of the pool-signed DMG and the app in it, the bundle size budget, and the tested-digest upload. For a pool-signed build it records the final DMG's image digest and skips its own smoke. A macOS-signer build (fallback path) still runs the smoke here, as before.
publish-nightly needs launch-smoke-nightly. When verify-final-dmg-nightly recorded an image, the job's if: and the "Select the build to publish" step both require launch-smoke-nightly to have passed on that same image. Otherwise this run's build is not published. If a continued build was recovered, publish falls back to it, as it already does for a failed verify.
New scripts/ci/dmg_image_digest.py: the SHA-256 of a UDIF DMG's data fork, resource fork and block map (each with its length), read from the koly trailer. New scripts/ci/nightly-steps/launch-smoke-image.sh: the digest check and output.

Why the unsigned DMG is the right thing to smoke

fresh_home_launch_smoke.py --dmg mounts the DMG read-only, dittos the app out to a private root and execs its binary directly with a fresh HOME. It doesn't test the DMG's signature, Gatekeeper, quarantine or the DMG's staple. Those stay in verify-final-dmg-nightly (verify-dmg-artifact.sh, signer-final-dmg-checks.sh). So the smoke is testing the app bundle. The app in the unsigned DMG is already Developer-ID signed and has its notarization ticket stapled: signer-dmg-image staples it before create-dmg.

Signing a DMG (rcodesign sign) and stapling it append a code signature and ticket after the block map and rewrite the 512-byte trailer. They don't touch the image the app mounts from. Checked on two production runs:

run	unsigned DMG sha256	final DMG sha256	image digest (both)
41444184160	`2b60b5ad…`	`885f27f7…`	`8ff7e021…` (data fork 255,996,820 B and 57,744 B block map byte-identical)
41439348303	differ (by 14,689 B of signature + ticket)		data fork 256,018,096 B and 57,801 B block map byte-identical

The two DMGs have identical image bytes, so the volume and every file of the app in it are byte-identical. That covers CodeResources and every Mach-O, and therefore the cdhash. Publish enforces this per build: it compares the image digest of the DMG that was smoked with the final DMG's.

Performance

Metric: build end (build-nightly-app completed) to NEXT feed upload (publish-nightly "Upload nightly-next appcasts to R2" completed).

run	before	smoke step removed from the path	after (projected)	delta	harness
41432827903	24:49	92 s	23:17	-1:32	jobs API timestamps
41437704107	18:58	106 s	17:12	-1:46	jobs API timestamps
41439348303	18:49	78 s	17:31	-1:18	jobs API timestamps
41444184160	21:09	78 s	19:51	-1:18	jobs API timestamps

How the projection works: after signer-dmg-image ends, the old path was signer-dmg (186-215 s) plus verify-final-dmg-nightly (187-271 s, smoke included). The new path is max(signer-dmg + verify-final without the smoke, launch-smoke-nightly). Without the smoke, verify-final takes 109-193 s, so that branch finishes 301-405 s after signer-dmg-image. launch-smoke-nightly takes setup (~10-20 s) + tooling checkout (~6-8 s) + the unsigned DMG download (same size as the final DMG's: 22-96 s in these runs) + the smoke (78-106 s), about 127-199 s. That leaves roughly 150-200 s of slack before its runner wait would be on the critical path. So the saving is the smoke step's duration, 78-106 s per build. Blacksmith minutes per day: unchanged on the owned-pick path. On a Blacksmith fallback the one extra Mac job uses about 2-3 min per build, the same smoke minutes moved, plus one more setup and DMG download.

Routing

launch-smoke-nightly uses the exact runs-on expression of verify-final-dmg-nightly/signer-dmg-image: the pick-nightly-sign-runner owned pick for this attempt, the attempt-2 nightly slot, the picker's alternate/runner, MACOS_RUNNER_26, then Blacksmith macOS 6 vCPU, with the fork branch (macos-26). It holds no secret and no environment, has no claim step, and doesn't hold the whole Mac (the old smoke didn't either). It selects no Xcode (added to the Xcode-selection exemption list, like the post-publish smoke).

Risk

Gating semantics are unchanged. A crash on both attempts fails the job and the build is not published. cannot-run still exits 0 and never holds a build, as before. If the kill switch flips mid-run, it fails safe: if verify-final records an image while the parallel smoke was skipped, publish refuses.
The smoke can now overlap verify-final-dmg-nightly's static checks of the same bundle id. They may land on the same mini. Those checks launch nothing, and the smoke runs a private copy with its own HOME and holds the per-bundle defaults lock.
Follow-up (separate repo): the runner hook's job-class table has verify-final-dmg-nightly as light. Until it lists launch-smoke-nightly, the new job is admitted under the default (heavier) class, so on a busy shared mini it may wait for capacity. It still isn't refused, and the slack above absorbs a short wait.

Kill switch: set the repository variable NIGHTLY_NEXT_PARALLEL_SMOKE=off. launch-smoke-nightly is skipped, verify-final-dmg-nightly records no image and runs the smoke itself, and publish needs nothing new. That is exactly the old path.

The legacy NIGHTLY-identity copy is unchanged: it shares no smoke step.

Tests

New ParallelLaunchSmoke in tests/test_nightly_signer_jobs.py (wired in ci-guards). It covers: the image digest is unchanged by an appended signature/ticket and changes with any image or block-map byte, and non-DMGs are refused; the job needs signer-dmg-image (not signer-dmg) and has no secrets/environment; the job routes like verify-final; the kill switch is honoured; verify-final skips its smoke only when it recorded an image; publish's if: and needs; and the select script runs with a failed, skipped or wrong-image smoke and refuses this build. launch-smoke-image.sh refuses a DMG with the wrong digest.
Updated the pins: test_nightly_universal_build.sh and test_nightly_next_recovery.py (the publish if:), test_nightly_trusted_routing, test_nightly_sign_runner, test_nightly_dedicated_mini (route lists), test_nightly_next_launch_smoke (the pre-publish smoke job is allowed in publish's needs), and test_ci_macos_xcode_selection (exemption).
.github/nightly-step-coverage.json, warm-sources.json and runners.json have entries for the new job and step.
Passing: test_nightly_signer_jobs, test_nightly_delivery_concurrency (functions called directly), test_nightly_trusted_routing, test_ci_runs_on_fallback, test_nightly_step_coverage, test_ci_fork_runner_routing, test_ci_warm_sources, test_runner_routing_table, test_nightly_next_launch_smoke, test_nightly_next_recovery, test_nightly_sign_runner, test_nightly_dedicated_mini, test_ci_signing_keychain_is_deleted_before_bundle_code, bash tests/test_nightly_universal_build.sh, bash tests/test_ci_self_hosted_guard.sh, and the other 145 tests that read nightly.yml or these tables. The three that fail (test_check_ghostty_zig_workflows, test_ci_cloud_probe_rescue, test_nightly_next_arm64_only) fail the same way on base.
actionlint 1.7.12-manaflow.1: the same 6 pre-existing info findings as base, none new. nightly.yml is 484,939 bytes (budget 490,000).

Agent session: agent session
🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

Runs the fresh-home launch smoke of pool-signed nightly-next builds in its own Mac job, on the unsigned DMG signer-dmg signs, while signer-dmg signs, notarizes and staples that DMG, instead of afterwards inside verify-final-dmg-nightly. That cuts about 80-105 seconds off every nightly-next build's critical path, and the smoke gate still holds: publish-nightly needs the new job and publishes only when the smoke passed on the same DMG image.

Signing and stapling a DMG only append bytes, so the image the app mounts from is identical before and after, which dmg_image_digest.py verifies per build.
launch-smoke-nightly takes the unsigned DMG signer-dmg-image built, verifies it is the exact file signer-dmg will sign, and runs the existing launch-smoke-gate.sh.
verify-final-dmg-nightly keeps every final-DMG check but skips its own smoke for pool-signed builds and records the final image digest.
The macOS-signer fallback and recovered builds are unchanged.
Set the repository variable NIGHTLY_NEXT_PARALLEL_SMOKE=off to put the smoke back inside verify-final-dmg-nightly.