#26534 ci(rescue): move every nightly.yml sign, verify and DMG job off a stalled Blacksmith macOS label

sr2's order: "Rescue gap on nightly-next: ci-owned-pool-rescue's include_blacksmith_macos covers cmux-next.yml only, so a nightly.yml job stalled on a Blacksmith macOS label is never moved. Today run 40585960129 had signer-verify queued on blacksmith-6vcpu-macos-26 for 3 h+, and 40610802228's sign job has been queued since 01:16Z. Please extend include_blacksmith_macos (or the equivalent) to nightly.yml's sign, verify and dmg-image jobs. That's the queue bound for the Linux signing path, since GitHub has no queue timeout."

Leo, 2026-10-27: "a waiting nightly, promote or nightly sign/verify job is admitted on a mini ahead of every PR, DEV and side job"; the routing order ends at Blacksmith macOS ("IT WAS ALWAYS FUCKING LAST RESORT"), and no job waits on one reserved machine.

Problem

For nightly.yml, the code that does what include_blacksmith_macos does for cmux-next.yml is the signing-stall rescue (NIGHTLY_SIGN_ALTERNATES, signing_stall()). It had three gaps:

1. signer-dmg-image and signer-verify were not in the table. In run 40585960129 (attempt 2), signer-verify sat queued on blacksmith-6vcpu-macos-26 from 00:34 to 01:52Z. Because it was not in the table it counted as unfinished work, so the run's stalled Verify final nightly DMG (arm64) was not moved either.
2. Main's classic nightly was not covered at all. Only nightly-next was. Main's runs-on uses its own alternate-label expression (pick-classic-sign-runner, which also checks the attempt that made the pick).
3. No test tied the table to nightly.yml. A job added or renamed in nightly.yml silently dropped out of the rescue.

The coordinator added a fourth item for this file, a blocking finding on #26494: main's prepare-nightly-variant legs will be pinned to the trusted mini right after the build. Main's push nightlies run in parallel, so a newer push's build can take that mini first. The rescue never moved a job waiting on a reserved runner, so the legs would sit behind a full build with no fallback.

Change (scripts/ci/owned_pool_rescue.py, scripts/ci/owned_pool_sources.py)

• NIGHTLY_SIGN_ALTERNATES gains signer-dmg-image and signer-verify. Each entry now takes a tuple of alternate-label expressions, so main's classic expression counts too.
• Target.signs_late: main's nightly attempt 1 gets the same signing-stall watch as nightly-next. Main's later attempts are unchanged, because a full re-run there builds the app again.
• A job pinned to a nightly's reserved runner that is not the app build (reserved_follower()) now waits at most RESERVED_FOLLOWER_SECONDS (5 min). After that it moves like a stuck job: the run is cancelled, then its failed jobs are re-run, and the re-run takes the Blacksmith fallback. The build's own wait stays untimed. A started job is still never cancelled: if any job of the run is running, the follower keeps waiting.
• pull_moved(): a main nightly whose app build succeeded (app_built()) is no longer treated as moot when main moves on. It used to be cancelled without a re-run, which threw away the build. Now its failed jobs are re-run. A newer run pending behind it in a shared group still wins, as before.
• Cancelling a started nightly still goes only through the rescue's existing path: signing_stall() acts only when every other job of the run has finished, and started_nightly_work() / nightly_cancel() re-check right before the cancel.

Test: derived from nightly.yml, not hard-coded

tests/test_ci_owned_pool_sources.py::test_every_nightly_sign_verify_and_dmg_job_on_blacksmith_macos_is_rescued reads main's rescue table (AST, sign_table()) and checks it against both branches' nightly.yml (the working tree plus origin/feat-cmux-next, fetched under Actions as the file already does). It finds every job whose id matches sign, verify or dmg and whose runs-on can name a Blacksmith macOS label (a literal or vars.MACOS_RUNNER_26; the CI_CLOUD_OVERFLOW_SAVED outage literal is ignored). It fails when such a job is missing from the table, or when its runs-on has none of the table's alternate-label expressions. Mutation check: removing signer-dmg-image from the table fails the test for origin/feat-cmux-next. The test already runs in ci-guards ("Validate that the persistent-pool rescue watches every owned-pool workflow"), so it is blocking on main.

#26491 (merged) removed signer-verify from feat-cmux-next. The derived list follows that change automatically; the table entry stays for runs still in flight.

Proof (replay of run 40585960129's real jobs at 01:44Z, head efc7808a's nightly.yml)

• main: watch | Verify final nightly DMG (arm64) queued on blacksmith-6vcpu-macos-26 ... but signer-verify not finished; cancelling the run would lose that work, so it is not rescued on this look
• this branch: stalled | Verify final nightly DMG (arm64), signer-verify queued on blacksmith-6vcpu-macos-26 with no runner past the 600s Blacksmith stall time, and every other job of the run has finished

Tests

• tests/test_ci_owned_pool_rescue.py: 335 pass. New cases: signer jobs are rescued; main's attempt 1 with the classic alternate is rescued; main moving on no longer drops a built run; a pinned follower moves at 5 min while the build never does; a follower never moves while a leg is running.
• tests/test_ci_owned_pool_sources.py: 11 pass, against the live origin/feat-cmux-next.

Deploy: the rescue checks out main, and a running sweeper hands over within CODE_CHECK_TICKS once main's copy changes, so merging deploys it.

Risk: medium. A main nightly whose sign or verify jobs stall now gets cancelled-then-re-run once nothing else in it runs. A main nightly's pinned follower moves to Blacksmith after 5 min.

Agent session: agent session

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Extends the nightly signing-stall rescue so every sign, verify, and DMG job in nightly.yml can be moved off a stalled Blacksmith macOS label, and covers main's classic nightly for the first time.

• NIGHTLY_SIGN_ALTERNATES gains signer-dmg-image and signer-verify; each entry now holds several alternate-label expressions, including main's classic one.
• Main's nightly attempt 1 gets the same signing-stall watch that nightly-next had.
• A job pinned to a nightly's reserved runner other than the app build waits at most 5 minutes before it moves like a stuck job (run cancelled, failed jobs re-run). The build's own wait stays untimed, and a started job is never cancelled.
• A main nightly whose app build succeeded is no longer dropped when main moves on: its failed jobs are re-run instead. A newer run pending behind it in a shared group still wins.
• A new test derives the covered jobs from both branches' nightly.yml, so a job added or renamed there fails until the rescue covers it.

Summary by CodeRabbit

• Bug Fixes
  • Nightly build monitoring now continues to watch queued app builds while identifying reserved-runner jobs that remain stuck for more than five minutes.
  • Signing-stall detection now covers additional signing, verification, and disk image jobs across both nightly branches, including supported alternate runner labels.
  • Nightly runs can continue checking later attempts, and a successful app build is no longer treated as a reason to skip further monitoring.