ci(nightly-next): a re-run that picks again signs the legacy identity on an owned Mac first

TE
7d ago
+111-19

What broke

NEXT nightly runs 41077184190 and 41094107023 (2026-10-30) never signed the cmux NIGHTLY (legacy) identity, so the old feed got no build from either one. A newer nightly tip did not cancel them. The owned-pool rescue (scripts/ci/owned_pool_rescue.py, signing-stall path) did, and only after every other job had finished:

  1. Attempt 1: sign-nightly-identity-legacy took a drained-mini pick on cmux12s and was refused there (glaeda drain bound). Its Blacksmith twin then queued on blacksmith-6vcpu-macos-26 with no runner past the 600 s stall time. The rescue cancelled the run and re-ran its failed jobs.
  2. Attempt 2 (re-run of failed jobs): the sign went to the alternate, blacksmith-12vcpu-macos-26, which was also stalled. The rescue cancelled again and re-ran in full.
  3. Attempts 3 and 4 (full re-runs): pick-nightly-sign-runner was skipped, because its if needs prepare-nightly-variant success and a full re-run of a published build builds nothing. The sign's owned route and its twin read the pick only when github.run_attempt == 1. So the sign went straight back to stalled Blacksmith macOS every time, and an owned Mac was never offered again.

The routing order (Linux, then owned Macs, then Blacksmith last) was skipped on every re-run.

What changes (.github/workflows/nightly.yml, scripts/ci/signing_runner.py)

  • pick-nightly-sign-runner also runs when only the legacy sign can run: decide succeeded, the ref is nightly-next, and NIGHTLY_NEXT_LEGACY_IDENTITY is not off. Every other job that needs it still gates on the app build, as checked against each dependent's if and needs.
  • Its "Pick a drained owned Mac for the signing jobs" step runs on any attempt, with PICK_ON_RERUN: "1". signing_runner.py owned still refuses a re-run unless that opt-in is set, so release.yml, the v0.64.25 repair and cmux-next-dev-daily keep their attempt-1-only behaviour.
  • sign_owned follows the same rule as the verify jobs' owned: attempt 1 of a push, or any attempt once the rescue sweeper's owned-pool-watch marker is uploaded. That means the sweeper watches the re-run.
  • sign-nightly-identity-legacy and its twin now read the pick under needs.pick-nightly-sign-runner.outputs.attempt == github.run_attempt instead of github.run_attempt == 1. This is the form the verify jobs already use. A re-run of failed jobs keeps the older pick (attempt 1), so the clause is false and the sign takes the alternate Blacksmith label as before. A re-run that ran the picker again takes the owned Mac first, and the twin signs if that Mac refuses it.
  • The rescue still finds the attempt-2 alternate it requires (github.run_attempt > 1 && needs.legacy-identity-target.outputs.sign_alternate) in the sign's runs-on, so its sign_stall_gate is unchanged.

Tests (tests/test_ci_signing_owned_mac.py, tests/test_nightly_universal_build.sh)

  • The owned-routing guard accepts a pick read either as github.run_attempt == 1 && pick && fromJSON(pick) or as needs.<picker>.outputs.attempt == github.run_attempt && .... The second form is allowed only when the picker's attempt output is exactly ${{ github.run_attempt }}. The twin's if must use the same clause as the sign's runs-on.
  • LegacySignReRunTakesAnOwnedMacFirst:
    • on attempts 1, 2 and 3 a fresh pick puts the sign on the owned Mac, and a refused sign gets its twin;
    • a re-run of failed jobs (pick from attempt 1, run at attempt 2) takes the Blacksmith alternate and no twin;
    • the picker runs when prepare-nightly-variant is skipped, but not on main and not with the legacy identity off;
    • sign_owned keeps the same marker rule as owned.
  • signing_runner.owned_pick on attempt 3 offers a Mac with PICK_ON_RERUN=1 and none without it.
  • Ran locally: test_ci_signing_owned_mac (all pass except test_every_listed_hook_is_one_glaeda_released_with_the_check, which reads glaeda's live releases: glaeda's newest hook 5d5ce6e8 was released minutes ago and the list sync PR adds it), test_workflow_file_size (nightly.yml at 489,721 bytes, under the 490,000 cap), test_nightly_trusted_routing, test_ci_owned_pool_sources, test_nightly_sign_runner and test_ci_owned_pool_rescue.

Risk

  • A full re-run now tries an owned Mac for the legacy sign before Blacksmith. If that Mac refuses, the twin signs on Blacksmith in the same attempt, as on attempt 1.
  • The picker job now also runs on runs that only continue a pending legacy submission. It is a Linux job of about 20 s with continue-on-error.
  • Not changed here: the rescue's cancel and re-run cycle when every Blacksmith macOS signing label is stalled. With this PR the cycle at least reaches an owned Mac on a full re-run.

Agent session: agent session

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes nightly re-runs so the legacy identity sign takes a picked owned Mac first instead of going straight to a stalled Blacksmith macOS label, which left the old feed without builds in runs 41077184190 and 41094107023.

  • The picker now runs when only the legacy sign can run, and picks on any attempt for the signing jobs.
  • The sign and its twin read the pick only on the attempt that made it, so a re-run of failed jobs keeps the Blacksmith alternate as before.
  • sign_owned follows the same marker rule as the verify jobs' owned, and other workflows keep their attempt-1-only behavior unless they set PICK_ON_RERUN.