ci(nightly): sign the cmux NIGHTLY-identity copy with the release signer pool, beside cmux NEXT

Merged
TE
5d ago
+588-53

Stacked on #32123: this branch's first commit is that PR's head. Land #32123 first; this diff is the second commit.

The old feed (files-next.cmux.com/nightly-next/appcast-arm64.xml, read by existing com.cmuxterm.app.nightly installs) ran 20-50 min behind cmux NEXT. Three causes:

  • The copy started only after publish-nightly (#25957).
  • Its macOS signer (sign-nightly-identity-legacy) holds a whole owned Mac. glaeda's exclusive admission kept it in "Set up runner" for 12.4 min (run 41416268105), 12.3 min (run 41415054395) and 33 min (run 41412703950). Runs 41413478144 and 41414248977 were still waiting at 10:05Z, 50 and 28 min in.
  • After that wait, most of those signs skipped: by then a newer NEXT build had replaced theirs.

This PR signs this run's own build with the release signer, alongside cmux NEXT's own signer jobs. It uses the same release-next secrets signer-app already uses on GitHub-hosted Ubuntu, plus APPLE_NIGHTLY_PROVISIONING_PROFILE_BASE64, which the macOS legacy signer already reads from that environment. Trust, secrets, environments and workflow permissions are unchanged.

What changed:

  • legacy-identity-target waits for prepare-nightly-variant, not publish-nightly. Its feed compare already accepts this run's prepared build as a candidate (CURRENT_BUILD_READY, source_run).
  • New legacy-signer-app (signer route, release-next). It takes this run's prepared app and stamps the cmux NIGHTLY identity (nightly_next_identity.py --legacy: the values nightly-next-legacy-identity.sh sets on a Mac). It embeds the NIGHTLY Developer ID profile after checking the profile's App ID and Developer ID, and embeds the verified SSH daemon manifest. Then it signs with cmux.nightly.entitlements in helper-first mode, and notarizes and staples.
    • It runs only for action new with source_run set to this run and the build equal to this run's.
    • It does not run when NIGHTLY_LEGACY_REQUIRED_COMMITS is set or the notary is paused.
  • New legacy-signer-dmg-image (secret-free owned Mac, the same route as verify-nightly-identity-legacy). It staples the app and runs Apple's checks (signer-apple-checks.sh). New legacy-pool-check-app.sh then checks the cmux NIGHTLY identity of the signed app: application-identifier and keychain group 7WLXT3NR37.com.cmuxterm.app.nightly, the profile, the build, the Sparkle key and the channel metadata. Last, it builds the unsigned DMG.
  • New legacy-signer-dmg (signer route, release-next) signs, notarizes and staples the DMG. It then hands it on under the macOS signer's names: artifact cmux-nightly-identity-signed and the outputs signed_ok, skip, notarize, apple, dmg_sha256, minimum_system_version and build_sha. Those outputs are set only by its last steps.
  • New verify-nightly-identity-legacy-pool runs the same steps as verify-nightly-identity-legacy on that DMG. It does not wait for the macOS signer, which is only skipped after publish-nightly.
  • sign-nightly-identity-legacy now needs publish-nightly, so the only job that holds an owned signing Mac still runs after publication, as #25957 requires. It runs only when legacy-signer-dmg did not sign. That covers a pool failure, a continued Apple submission, another run's build, and a required-commit check.
  • publish-nightly-identity-legacy and nightly-identity-legacy-published read the pool's outputs first, then the macOS signer's. Publication still requires publish-nightly to have published this exact build.
  • NIGHTLY_NEXT_LEGACY_POOL=off turns the new path off.

Tests:

  • New LegacyPool cases in tests/test_nightly_signer_jobs.py: the gate, secrets confined to the two hosted jobs, the fallback wiring, outputs set only at the end, the merged publication inputs, and the pool verify having the same steps as the legacy verify.
  • A --legacy case in tests/test_release_signer_nightly.py.
  • test_nightly_next_post_publish.py now pins the macOS signer, not the target, after publication, and pins the pool jobs to the signer route.
  • Routing and guard registries updated: runners.json macos_jobs, warm-sources.json, step coverage, the upload allowlist, RELEASE_POOL_SIGNERS, owned-route lists, and the Blacksmith-only debt list (the signer route's hosted exception).

Performance

metricbeforeafterdeltaharness or run link
Old-feed lag behind cmux NEXT, publish-nightly end ā†’ old feed published21.5 min with no runner wait (run 41417028889: 09:42:17 ā†’ 10:03:48); 30-50+ min with the exclusive-Mac wait (runs 41412703950, 41413478144, 41414248977)~1-3 min expected (the pool finishes beside NEXT's chain; publication waits only for publish-nightly and the pool verify)about -19 to -48 minestimate from run 41416268105 stage times; first nightly-next run after merge confirms
macOS signer "Set up runner" (exclusive admission) on the old feed's path12.3-33 min, or still waiting at 50 min0 when the pool signs (that job is skipped)-12 to -50 minruns above
Owned Mac minutes held exclusively per run~12-35 min0 when the pool signs-12 to -35 minruns above
Pool stage estimates (from cmux NEXT's own pool jobs, run 41416268105)n/asign + helper notarization ~2.1 min, app notarization ~3.2 min, image ~5 min, DMG ~3.6 min, verify ~4.5 minn/asigner-app, signer-app-notary, signer-dmg-image, signer-dmg, verify-nightly-identity-legacy

Expected timeline (run 41416268105's times): prepare-nightly-variant ends 09:15:57, the target ~09:16:30, the signed and notarized app ~09:22, the image ~09:27, the DMG ~09:31, the verify ~09:35:30, publication right after publish-nightly (09:34:57). Before, this run's build never reached the old feed: its sign waited 12.4 min for the Mac and then skipped because a newer build had replaced it. The next build reached the old feed at 10:03.

Risk: this is the first release-signer signature that uses a provisioning profile (the NEXT identity ships without one). sign-app.sh already reconciles a bundle's embedded profile through reconcile-entitlements-with-profile.py, the same script sign-cmux-bundle.sh uses on a Mac. The image job reads the signed entitlements back with codesign -d before anything is packaged. Any failure falls back to today's macOS signer in the same run, and the publish step still publishes only a DMG whose sha256 the verify job tested.

Validation, run locally: all 139 tests that read nightly.yml, the runner and warm-source registries, the step coverage, the upload allowlist, nightly-steps or release-signer were run. Six fail, and each fails the same way on base: test_check_ghostty_zig_workflows, test_ci_blacksmith_linux_failover, test_ci_blacksmith_linux_sizes, test_ci_cloud_probe_rescue, test_ci_linux_single_size, test_nightly_next_arm64_only. legacy-pool-prepare.sh was also run against a stub build tree with a CMS-signed test profile. It wrote the cmux NIGHTLY plist (name, bundle id, cmux-nightly scheme, old feed URL, track key, build, commit, no callback override) and refused a profile whose App ID is the NEXT one. shellcheck and the bash 3.2 parse guard pass.

Prior art: #24987 first overlapped the compatibility notarization with NEXT's. #25957 moved it after publication so the owned-Mac signer would not take the signing cap, and that holds for the macOS signer. #27689 skips a build NEXT has passed. #28583 lets any run publish the copy. #29678 makes the feed watch re-run a stale copy. #31831 put the signer jobs on hosted Ubuntu first.

Agent session: agent session

šŸ¤– Generated with Claude Code

Summary by cubic

Signs the cmux NIGHTLY-identity copy for com.cmuxterm.app.nightly with the release signer pool, cutting the old feed's lag behind cmux NEXT from 20-50 min to an expected 1-3 min. Previously the compatibility copy waited for publish-nightly, then for an exclusive owned Mac that often sat in "Set up runner" for 12-33 min, by which point a newer NEXT build had replaced it and most signs were skipped. Now the copy is signed from this run's prepared app, beside cmux NEXT's own signer jobs.

  • legacy-identity-target starts from prepare-nightly-variant (this run's prepared build), not publish-nightly.
  • New jobs legacy-signer-app, legacy-signer-dmg-image, legacy-signer-dmg, and verify-nightly-identity-legacy-pool sign, notarize, build, and verify the DMG on the signer route plus one secret-free Mac; publication reads whichever signer signed.
  • The macOS signer stays the fallback, still after publish-nightly (#25957), covering pool failures, continued submissions, other runs' builds, and required-commit checks.
  • Disable with NIGHTLY_NEXT_LEGACY_POOL=off.