ci(nightly-next): wait 20m for Apple, then hand the build to recovery
Summary
This PR stops one stalled notarization from freezing the nightly-next feed.
- The problem: nightly-next runs one build at a time (a single concurrency group). Each build's sign job waited up to 40 minutes for Apple before handing a still-pending submission to recovery. On 2026-10-14, submission
5912557estayedIn Progressat Apple for over 90 minutes, which held every newer build behind it for 40 minutes per run. - The change: published nightly-next now waits 20 minutes. A build still in Apple's queue goes to the recovery artifact, and the run frees the group for the next build. Main, RC and notary-test runs keep their 40-minute and 160-minute waits.
- Recovery already publishes the newest accepted build.
find-nightly-next-recovery.pylists pending builds newest first, andrecover-nightly-next-notarizationpublishes the first one Apple accepted. A stalled submission never hides a newer accepted one. - Recovery keeps checking. It runs in every nightly-next run, and every green feat-cmux-next Release compile requests a continuation (
continue-nightly-next.sh, at most every 30 minutes). Since #20509, those jobs run onblacksmith-6vcpu-macos-26, so they no longer queue behind PR CI.
History: #18736 raised the wait from 20 to 40 minutes after three slow builds in a row published nothing. That was before newest-first recovery existed. Now a slow build is handed off instead of lost.
Verification
tests/test_nightly_universal_build.sh was updated in a separate first commit:
- Red: against base's workflow it fails with "nightly-next bounded notarization contract is missing: … '20m' || '40m'".
- Green: it passes with this change.
Also run:
tests/test_notarize_nightly_dmg.shpasses. It checks that the step and job timeouts still cover the wait.actionlint .github/workflows/nightly.ymlpasses.
Changelog
none
🤖 Generated with Claude Code
Need help on this PR? Tag @codesmith-bot with what you need.
Summary by cubic
Reduces the notarization wait for published nightly-next builds from 40 minutes to 20 minutes so one stalled Apple submission no longer holds every newer build behind it. Nightly-next runs one build at a time, and a submission that stayed In Progress at Apple for over 90 minutes froze the feed on 2026-10-14.
- Recovery now considers the 5 newest pending builds so an Invalid one cannot hide an older Accepted build behind the previous limit of 3.
- Published nightly-next hands a still-pending submission to the recovery artifact after 20m; main, RC, and notary-test runs keep their 40m and 160m waits.
- Recovery publishes the newest accepted build and keeps running on every nightly-next run, so a slow build is handed off rather than lost.
- Updates the workflow contract test to expect the 20m wait for published nightly-next.