#18553 fix(ci): resume timed-out nightly notarization

Changelog

Fixed nightly DMG publication when Apple's notarization service takes longer than the CI wait window.

A 40-minute notarytool --wait timeout can still mean Apple accepted the upload and is processing it. The workflow now records the exact submission ID, DMG hash, release inputs, and notary output, uploads a recovery artifact, and resumes that submission from a separate workflow. The continuation verifies the source run, archive contents, DMG hash, ticket status, stapling, and publication identity before it can publish. It never submits the same DMG a second time.

The fast internal dogfood path remains available with notarization skipped, and the normal publishing path still fails closed until Apple reports Accepted.

Validation:

• bash tests/test_notarize_nightly_dmg.sh
• python3 tests/test_notarization_recovery_manifest.py
• bash tests/test_nightly_universal_build.sh

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Fixes nightly DMG publication when Apple's notarization service takes longer than CI's wait window. Published builds now submit the DMG without waiting, and a continuation workflow resumes that exact submission, publishing only after Apple reports Accepted—never re-submitting the DMG.

Recovery flow

• The build lane records the submission ID, DMG hash, signed app, and notary evidence, and uploads them as a recovery artifact.
• An automated workflow_run continuation (with a manual single-variant dispatch fallback) verifies the source run, recovery contents, DMG hash, ticket status, stapling, code-signing checks, and publication identity, then publishes the DMGs, Sparkle feeds, and R2 objects and moves the channel tag.
• The continuation skips private or non-publishing nightly runs, and stale-release guards stop an older build from replacing a newer release.
• Publication fails closed until Apple reports Accepted, and the fast dogfood path still skips notarization.
• Recovery artifact retention is extended to 14 days and publication races are handled with per-channel, per-variant concurrency locks.

---

[!NOTE]
High Risk
Changes release publishing, GitHub/R2 feeds, and channel tag moves to a new workflow_run path with Apple notarization handoff; mistakes could ship wrong builds or skip publication, though strict manifest/SHA/stale guards mitigate that.

Overview
Nightly macOS publication no longer blocks on Apple’s notarization queue in the build job. For publishable channels (should_publish), notarize-nightly-dmg.sh can run in submit-only mode (CMUX_NOTARY_SUBMIT_ONLY): it uploads the DMG, records submission ID/SHA and release metadata in sidecars, sets submission_pending, and exits without stapling. The nightly matrix then builds a recovery bundle (signed DMG, state/log, tarred app, cmux-nightly-notarization-recovery.json) and uploads per-variant artifacts with longer retention; in-build Sparkle feeds, syspolicy gating, and the old publish-nightly path are skipped when publishing is deferred.

Continuation is a new trusted pipeline replacing nightly-notarization-recovery.yml and recover-nightly-notarization.sh. auto-resume-nightly-notarization.yml runs after Nightly macOS build completes: preflight validates the source run/branch and all three recovery artifacts; wait-and-staple uses resolve-notarization-recovery.py + resume-nightly-notarization.sh to wait on the same submission (no re-submit), staple, and emit verified DMGs/feeds; publish assembles variants, guards stale releases, publishes GitHub/R2 assets, moves the channel tag, and closes failure issues; optional delta generation/republish runs afterward. resume-nightly-notarization.yml adds manual workflow_dispatch recovery for one variant.

CI guards shift to tests/test_notarization_recovery_manifest.py and expanded workflow contract checks in test_nightly_universal_build.sh / test_notarize_nightly_dmg.sh.

^{Reviewed by Cursor Bugbot for commit 071efe582471bab2f1d47de50a7f1d536e7b4a80. Bugbot is set up for automated code reviews on this repo. Configure here.}

Summary by CodeRabbit

• Reliability
  • Nightly macOS releases can resume notarization after delays or failures without rebuilding or resubmitting. Recovery checks the build and notarization results before publication.
  • Notarization can continue asynchronously, with recovery evidence retained for up to 14 days.
  • Releases publish only after all variants pass verification, with safeguards against stale or mismatched builds replacing newer releases.
  • Eligible releases can recover automatically or through manual recovery. Delta updates are published when available; otherwise, full downloads remain available.