ci(nightly-next): retry only signing and publish of a failed nightly from its saved unsigned app

Merged
TE
8d ago
+482-1

Leo: "WHAT FUCKING FAILED RUNS HOW THE FUCK CAN YOU NOT JUST FUCKING INCREMENTALLY FUCKING RETRY WHTA HTE TUFK WE ALREADY FUCKING DEALT WITH INVESTIGATING HTE FUCKING CVACEHS WHY THE FUCK ARE WE ALWAYS REINVESTIGATING THIS IS THE FUCKING META QUESTION WHER ETHE FUCK ARE OUR FUCKING NOTES"

What

When a nightly-next run's signing, notarization, verification or publication fails after a good compile, the scheduled feed watch now retries only those steps. It does this by dispatching nightly.yml on nightly-next with nightly_next_unsigned_run / nightly_next_unsigned_attempt (#21161), which signs and publishes the saved unsigned app without compiling. Before this, only a person could dispatch that, so a failed signing waited for the next tick's full new build.

New step in nightly-next-feed-watch.yml (continue-on-error, after the continuation step): scripts/ci/nightly_next_unsigned_resume.py. On each 30-minute check it dispatches at most one resume, for the newest nightly-next push run that meets all of these:

  • it completed with failure or timed_out. A cancelled tip is left to the tick (nightly_next_cancelled_tip.sh);
  • build-nightly-app and build-nightly-ghostty-cli-helper both succeeded in the same attempt;
  • its build (run id plus the two-digit producer attempt) is above the newest build in the NEXT feed, so publication would not refuse it as superseded;
  • its unsigned inputs are still kept (cmux-nightly-unsigned-app is kept for one day), and it holds no cmux-nightly-notarization-recovery-* artifact. A recovery artifact is continue-nightly-next.sh's to publish once Apple accepts;
  • no resume is queued or running, and this run has had fewer than NIGHTLY_NEXT_MAX_UNSIGNED_RESUMES resumes (default 2; 0 turns this off).

Resumes are counted by title. The companion PR on feat-cmux-next (dep-fleet/nightly-resume-run-name) gives them the title Resume nightly-next run <id> attempt <n>. Until nightly-next's nightly.yml carries that title, this step only reports and dispatches nothing.

Nothing is cancelled. A resume runs in its own concurrency group (run-<id> on nightly-next), beside any build of the newer tip, and publication stays latest-wins. The dispatched run re-validates its source itself before signing (resume-unsigned-nightly.py plan). The step reads one run listing, plus jobs and artifacts for at most 3 failed runs newer than the feed. A run older than the feed costs no reads.

Decision on the current runs (dry run, read-only)

Feed newest build 4073801205901. The two recent failures, runs 40735210057 and 40724946461, both failed in publish-nightly ā€ŗ "Recheck the recovered build against both feeds" with nightly_version: build 4073521005701 is not above 4073801205901. A newer build had already published, so a resume would be refused too. Decision: nothing to resume, with no extra reads.

Risk

Medium-low. The step is continue-on-error. It writes only one workflow_dispatch and uses the actions: write the job already has. Repeated dispatches are bounded three ways: one resume at a time, a per-run cap, and the one-day lifetime of the unsigned app. It does not touch runs-on.

Testing

  • python3 tests/test_nightly_next_unsigned_resume.py: 16 tests. They cover the newest-run choice, superseded builds, a failed compile, mixed producer attempts, a re-run producer attempt, cancelled and live runs, one resume at a time, the per-run cap falling through to an older run, expired inputs, a pending notarization, the title gate, the off switch, the read bound, and the workflow step. Added to ci-guards' release-notary group.
  • python3 tests/test_nightly_next_feed_watch.py (77), tests/test_nightly_delivery_concurrency.py and tests/test_nightly_promotion_admission.py pass. actionlint is clean on the feed watch.

šŸ¤– Generated with Claude Code

Agent session: agent session

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by cubic

Retries the signing and publication of a failed nightly-next run from its saved unsigned app instead of waiting for the next full build.

Until now only a person could dispatch such a resume; a run that failed after a good compile at signing, notarization, verification, or publication came the next tick's whole new build. The feed watch now dispatches nightly.yml on nightly-next with nightly_next_unsigned_run / nightly_next_unsigned_attempt, which signs and publishes the saved app without compiling.

Details

  • Adds a continue-on-error step running scripts/ci/nightly_next_unsigned_resume.py to nightly-next-feed-watch.yml. Each 10-minute check dispatches at most one resume for the newest failed push run whose two producer jobs succeeded ā€” naming the newest attempt where both did ā€” whose build is newer than the newest build in the NEXT feed, and whose unsigned artifacts are still kept. Every failed run newer than the feed is judged, so a resumable run behind newer failed compiles is still found; runs older than the feed cost no reads.
  • Skips timed-out runs ā€” the resume validator refuses a timed-out source ā€” and runs that hold a notarization recovery artifact; the continuation publishes those once Apple accepts. Resumes are capped at NIGHTLY_NEXT_MAX_UNSIGNED_RESUMES (default 2; 0 disables), counted by title over the unsigned app's one-day lifetime, so nothing dispatches until nightly.yml on nightly-next titles them.
  • Nothing is cancelled: a resume runs in its own concurrency group beside the newer tip's build, publication stays latest-wins, and the dispatched run re-validates everything before signing. A running resume no longer counts as carrying the tip ā€” in either carries_tip or next_carries_tip ā€” so it cannot hold back a build of the newer tip.
  • On the current runs the decision is nothing to resume: the two recent failures were superseded by a newer published build, so a resume would be refused too. 16 unit tests added to ci-guards' linux-guard lane.

Summary by CodeRabbit

  • Release Improvements
    • Failed nightly-next runs that complete the build but encounter signing or publication errors can retry those steps using the saved build, without rebuilding.
    • Retries are limited to the configured maximum, and another retry waits while one is queued or running. Setting the maximum to zero disables retries.
    • Nightly feed monitoring excludes retry runs when checking whether a run includes the latest changes.