#33109 nightly: spend no notarization on a build a newer published build is above

Apple caps notarizations per day. On 2026-11-03 we hit the cap (429 RATE_LIMIT, "Exceeded daily limit for notarizations"), and every NEXT nightly from 03:46Z failed at a notary step for hours. About ten runs were signing side by side, and only the newest could ever publish: publication is latest-wins (nightly_version.py check). Each run spends a notarization each on the helper, app and DMG, for both feeds (#32140), so a run that is already overtaken burns up to six for nothing.

Change

• pick-nightly-sign-runner runs right before signing and, since #32966, on hosted Ubuntu. It gets a last step, "Leave notarization to a newer published build" (scripts/ci/nightly-steps/sign-pick-overtaken.sh).
  • The step reads the cmux NEXT feed. When the feed's highest sparkle:version is above this run's build (the run id plus a two-digit attempt), it sets overtaken=true. That is the same rule publish-nightly applies, so this build could only be called superseded there.
  • It runs only on nightly-next pushes, with continue-on-error, a 1-minute budget, and curl capped at 15 s.
• signer-app and the Mac fallback build-sign-notarize-nightly skip on overtaken, the way they already skip on resolve's superseded.
  • Everything after them (signer-app-notary, signer-dmg-image, signer-dmg, the verify jobs, publish-nightly) then skips as it does for a superseded run, with no red and no failure issue.
  • The legacy (cmux NIGHTLY) signer already signs only the newest NEXT feed build when it is this run's, so it needs no change.
• Only a published newer build counts, never a newer push alone. A newer run that then fails must not leave the feed with nothing; that is why #28961 turned off the ref-moved skip. An unreadable feed signs as before.
• The picker's job timeout goes from 15 to 16 minutes, keeping room for every step budget (an existing test checks this).

Both feeds stay as they are. The stapled app inside the DMG and both notarizations are unchanged. Consolidating to one feed is #33050, for later. #33028 (merged) waits out a 429 when one still happens.

Performance

metric	before	after	delta	harness or run link
notarizations spent by a run a newer published build is above	up to 6 (helper, app, DMG × 2 identities), plus the Mac fallback's retry	0	-6 per overtaken run	runs 41570134320 / 41568520110 job lists; numbers per day to follow in a comment
wall time added to the feed's path	0	one feed read (< 2 s) inside the existing picker job	~0	no new job
Blacksmith minutes/day	unchanged	unchanged	0

Tests

tests/test_nightly_sign_runner.py gets an Overtaken class:

• a newer published build skips;
• an older or equal feed signs;
• the attempt is part of the build;
• an unreadable feed signs;
• the signers and the fallback are wired to the output, and the step can never fail the pick.

The new tests fail without the change. tests/test_nightly_step_coverage.py, the routing and signer suites, and tests/test_ci_self_hosted_guard.sh pass.

Agent session: agent session

🤖 Generated with Claude Code

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Stops nightly runs that are already overtaken by a newer published build from spending notarizations, which are capped by Apple per day (hit as a 429 RATE_LIMIT on 2026-11-03, failing every nightly for hours).

• The signing picker now reads the cmux NEXT feed and skips notarization when its highest build is above this run's, matching the rule publish-nightly applies.
• Only a published newer build counts, never a newer push alone, so a newer run that fails still leaves the feed with something; an unreadable feed signs as before.