ci(nightly-next): wait 20m for Apple, then hand the build to recovery

Merged
TE
23d ago
+13-11

Summary

This PR stops one stalled notarization from freezing the nightly-next feed.

  • The problem: nightly-next runs one build at a time (a single concurrency group). Each build's sign job waited up to 40 minutes for Apple before handing a still-pending submission to recovery. On 2026-10-14, submission 5912557e stayed In Progress at Apple for over 90 minutes, which held every newer build behind it for 40 minutes per run.
  • The change: published nightly-next now waits 20 minutes. A build still in Apple's queue goes to the recovery artifact, and the run frees the group for the next build. Main, RC and notary-test runs keep their 40-minute and 160-minute waits.
  • Recovery already publishes the newest accepted build. find-nightly-next-recovery.py lists pending builds newest first, and recover-nightly-next-notarization publishes the first one Apple accepted. A stalled submission never hides a newer accepted one.
  • Recovery keeps checking. It runs in every nightly-next run, and every green feat-cmux-next Release compile requests a continuation (continue-nightly-next.sh, at most every 30 minutes). Since #20509, those jobs run on blacksmith-6vcpu-macos-26, so they no longer queue behind PR CI.

History: #18736 raised the wait from 20 to 40 minutes after three slow builds in a row published nothing. That was before newest-first recovery existed. Now a slow build is handed off instead of lost.

Verification

tests/test_nightly_universal_build.sh was updated in a separate first commit:

  • Red: against base's workflow it fails with "nightly-next bounded notarization contract is missing: … '20m' || '40m'".
  • Green: it passes with this change.

Also run:

  • tests/test_notarize_nightly_dmg.sh passes. It checks that the step and job timeouts still cover the wait.
  • actionlint .github/workflows/nightly.yml passes.

Changelog

none

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Reduces the notarization wait for published nightly-next builds from 40 minutes to 20 minutes so one stalled Apple submission no longer holds every newer build behind it. Nightly-next runs one build at a time, and a submission that stayed In Progress at Apple for over 90 minutes froze the feed on 2026-10-14.

  • Recovery now considers the 5 newest pending builds so an Invalid one cannot hide an older Accepted build behind the previous limit of 3.
  • Published nightly-next hands a still-pending submission to the recovery artifact after 20m; main, RC, and notary-test runs keep their 40m and 160m waits.
  • Recovery publishes the newest accepted build and keeps running on every nightly-next run, so a slow build is handed off rather than lost.
  • Updates the workflow contract test to expect the 20m wait for published nightly-next.