ci(nightly-next): watch cmux NIGHTLY's feed too, and re-run the legacy job that left it behind

Merged
TE
25d ago
+269-9

Leo (2026-10-25 22:50Z): "but where are the nightlies? i dont see a new thing on the bottom left"

What happened

  • cmux NIGHTLY (com.cmuxterm.app.nightly, Leo's daily driver) has SUFeedURL = https://files-next.cmux.com/nightly-next/appcast-arm64.xml, the old feed (NIGHTLY_NEXT_LEGACY_FEED_BASE).
  • Only publish-nightly-identity-legacy in nightly.yml writes that feed: it re-signs each cmux NEXT Nightly build as cmux NIGHTLY (cx-x7hwx).
  • Every legacy job since 21:10Z failed on HTTP 403: API rate limit exceeded for installation, the Actions token's quota. Failing runs: 40234225207, 40236532331 and 40240049419 (sign), and 40235404801 (publish).
  • The old feed stayed at 4020374816001 (15:27Z) while cmux NEXT's feed (/nightly-next/next/) reached 4023540480101 (22:19Z). Leo was 7+ hours behind.
  • nightly_next_feed_watch.py only watched cmux NEXT's feed, so nothing alarmed.

Why not mirror /next/ onto the old path: /next/ items are com.cmuxterm.app.nightly.cmux-next builds signed with the cmux NEXT Sparkle key. cmux NIGHTLY would refuse them, or replace itself with a different app.

What changes (scripts/ci/nightly_next_feed_watch.py, runs every 30 min on main)

  • It also reads cmux NIGHTLY's feed and compares the newest Nightly-ring build on each (items tagged cmux-next, as nightly_next_ring.py select counts them).
  • When cmux NIGHTLY's feed has waited more than 15 minutes for the oldest cmux NEXT Nightly build it is missing (a newer publication never restarts the wait), the watch finds the nightly-next run that built it. The build number is the run id plus a two-digit attempt: 4023540480101 is run 40235404801.
  • If that run has a failed legacy job, the watch re-runs only the first one (POST /actions/jobs/{id}/rerun, which also re-runs its dependents; covered by the existing actions: write), up to 3 attempts. The job ids come from the nightly-next branch's nightly.yml, the copy those runs execute.
  • Either way, the scheduled run goes red with the reason: still running, attempts used up, nothing failed to re-run, or not a nightly-next run.
  • No new timer: it adds work to the existing 30-minute schedule, and stops at the attempt cap.

Tests: tests/test_nightly_next_feed_watch.py gains LegacyFeedTests:

  • the failed legacy job of the run that built the build is re-run;
  • a current or fresh (under 15 min) feed is left alone;
  • only Nightly-ring builds count;
  • a run that is still running, used 3 attempts, has no failed legacy job, or is off nightly-next is reported stale with no re-run;
  • a stale cmux NIGHTLY feed fails the scheduled run, even when the GitHub API errors;
  • a newer publication never restarts the wait.

All 29 pass locally. The static check that each app's SUFeedURL is the feed its publisher writes is #25177 (on nightly-next, where those jobs live).

Restore, measured: the re-run of run 40235404801 put 4023540480101 on the old feed at 23:15:13Z, and cosmic-gray's NIGHTLY updated to it at 23:17:13Z.

Not in this PR: the root cause is something exhausting the Actions installation quota and starving release jobs. That's CI's area, and claude-sr2 has the details.

No capture: no UI; a CI watcher change.

Risk: low.

  • Success: a legacy job a 403 kills is re-run within 30 minutes, and a stale cmux NIGHTLY feed shows red.
  • Failure: extra re-runs of a job that keeps failing, capped at 3 attempts.

Agent session: cosmic-gray

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by CodeRabbit

  • Chores
    • Nightly build monitoring now compares the legacy feed with the current feed and flags missing builds once they are delayed by at least 15 minutes.
    • For eligible nightly runs, monitoring can retry the earliest failed identity check, subject to the run’s retry limit.
    • The scheduled check reports legacy-feed and retry errors, and marks the check as failed when the legacy feed is stale—even if the current feed is also stale or a GitHub API error occurs.