fix(nightly-next): continue an earlier run's pending notarization

Merged
TE
1mo ago
+1018-50

Summary

Part (2) of the nightly-next continuation, after #18713. A published nightly-next build whose 20m in-job notarization wait runs out leaves a recovery artifact and is not published. Each later published nightly-next run now continues it, inside nightly.yml on nightly-next with the release-next environment, the nightly-next guard and the cmux-nightly-next bucket:

  • find-nightly-next-recovery (Linux, no secrets): scripts/ci/find-nightly-next-recovery.py lists up to 3 earlier completed nightly-next runs that hold a live cmux-nightly-notarization-recovery-arm64-* artifact, newest first. A run qualifies only while its build is above every build on both the nightly-next and main nightly feeds (the existing no-downgrade rule); the first one that is not ends the search.
  • recover-nightly-next-notarization (Mac, release-next):
    • Validates each candidate's manifest against its run and the exact DMG (check-manifest), and rechecks the feeds.
    • Asks Apple for the saved submission's status. This uses notarize-nightly-dmg.sh with the new CMUX_NOTARY_CONTINUE_STATE: no rebuild and no resubmission. A submission still In Progress stays pending, and Invalid fails.
    • For the newest Accepted one, staples and validates the app and DMG exactly as the build job does, generates the cmux-next appcast with the cmux-next key, and runs the guard.
  • publish-nightly gets a first step, "Select the build to publish":
    • It publishes this run's build when Apple accepted it, otherwise the recovered build. Every later step takes the build number, build commit, tip and distance from that step.
    • For a recovered build, it rechecks the feeds inside the publication lock. The daemon assets and the source archive come from the recovered variant.
    • Main's NIGHTLY and RC always select their own build, so they are unchanged apart from the job if now being explicit about !cancelled().
  • The recovery artifact now records the build commit, tip and distance, and carries the SSH daemon assets and (when published) the source archive. Their own artifacts expire after 3 days; the recovery artifact lasts 14.
  • Recovered publications skip delta generation and release-note signing. Both are best-effort enrichment.

Limits

  • Recovery artifacts written before this PR lack the build commit, so they are refused (check-manifest). 6d443005 is not picked up, as agreed.
  • The recover job validates the recovered app with this run's checkout. If the ghostty or ghostty-next gitlinks moved between the two commits, the license check refuses that build. It fails closed: the build is skipped, not published.
  • Not exercised end to end. That needs a nightly-next run whose own wait times out, followed by one that finds it.

Testing

  • Red commit 59cfdb1e800a: tests/test_nightly_next_recovery.py (13 tests at that commit) fails with the finder missing and the workflow unchanged. tests/test_notarize_nightly_dmg.sh fails continuing an Accepted saved submission ("Missing APPLE_SIGNING_IDENTITY", the script had no continue mode).
  • Green: test_nightly_next_recovery.py passes 15 tests: the finder, check-manifest, the select step run with own, pending, failed and skipped inputs, job gating and environments, and the recovery artifact contents. test_notarize_nightly_dmg.sh passes, covering continue mode Accepted, In Progress, Invalid and a DMG SHA mismatch.
  • The resolver test now exempts the two recovery jobs (they continue an earlier run's build) and accepts publish-nightly's selected build commit, while requiring that the select step's own build is the resolver's build_sha. The universal test pins the new publish-nightly if.
  • The whole ci-guards release-notary group passes locally, and cmux-next-guards (#18718) runs it on this PR. actionlint is clean. test_ci_fork_runner_routing adds no new nightly.yml findings versus base.

Changelog

none

🤖 Generated with Claude Code

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Summary by cubic

Fixes the nightly-next notarization workflow so a published build whose 20-minute wait times out is still published by a later nightly-next run, instead of being lost.

  • Adds find-nightly-next-recovery and recover-nightly-next-notarization to nightly.yml. The finder lists up to 3 earlier completed nightly-next runs holding a live recovery artifact, newest first, and only while their build stays above every build on both feeds.
  • The recover job validates the candidate's manifest and exact DMG, asks Apple for the saved submission's status without resubmitting or rebuilding, and staples and validates the newest Accepted one. It generates the cmux-next appcast and runs the nightly-next guard.
  • publish-nightly selects this run's build when Apple accepted it, otherwise the recovered build; every later step uses the selected build's number, commit, tip and distance. For a recovered build it rechecks the feeds inside the publication lock and uses the recovered daemon assets and source archive.
  • The recovery artifact now records the build commit, tip/distance, SSH daemon assets, and source archive, and lasts 14 days instead of 3.
  • Recovered publications skip delta generation and release-note signing.

Limits

  • Recovery artifacts written before this PR lack the build commit and are refused.
  • The license check fails closed if the ghostty or ghostty-next gitlinks moved between the two commits.
  • Not exercised end to end; requires a nightly-next run whose own wait times out, followed by one that finds it.

Testing

  • test_nightly_next_recovery.py covers the finder, check-manifest, and the select step with own, pending, failed, and skipped inputs.
  • test_notarize_nightly_dmg.sh covers continue mode for Accepted, In Progress, Invalid, and a DMG SHA mismatch.
  • Actionlint is clean and the ci-guards release-notary group passes.

[!NOTE]
High Risk
Changes nightly-next release publication, notarization handoff, and which commit/assets ship when recovery runs—mistakes could publish the wrong build or skip updates.

Overview
Adds nightly-next notarization recovery so a later published run can finish and ship a build whose in-job Apple wait timed out, instead of leaving users without an update.

When notarization stays pending, the recovery artifact now records build commit, tip, and lag, and bundles SSH daemon assets and optionally the source archive (14-day retention). New jobs find-nightly-next-recovery and recover-nightly-next-notarization scan prior runs for live recovery artifacts (only while the build is still above both nightly-next and main feeds), validate manifests, and notarize-nightly-dmg.sh gains CMUX_NOTARY_CONTINUE_STATE to poll an existing submission without rebuilding or resubmitting the DMG.

publish-nightly adds Select the build to publish: it prefers this run’s accepted build, otherwise publishes the recovered one; checkout, notes, tags, and artifact patterns follow that choice. Recovered publications skip delta generation and release-note signing and recheck feeds inside the publication lock.

CI guards and tests (test_nightly_next_recovery.py, extended notarize and resolver checks) lock in the workflow contract.

Reviewed by Cursor Bugbot for commit 58c02565108fadb9a4293fe471928c71720e2bc6. Bugbot is set up for automated code reviews on this repo. Configure here.