ci(nightly): sign, notarize and publish take a Blacksmith label with a moving queue

Merged
TE
11d ago
+333-17

The nightly's sign, notarize, verify and publish jobs now take a Blacksmith macOS label whose queue is moving, not vars.MACOS_RUNNER_26 alone.

Why. From about 01:20Z on 2026-10-26, MACOS_RUNNER_26 named blacksmith-12vcpu-macos-26, a label that started no job. Every nightly's signing jobs waited on it, and the nightly-next feed went about eight hours stale. The variable was flipped to 6vcpu by hand at 09:17Z. This change stops a stale variable from stranding the nightly again.

Coordinator's direction: "signing stays on Blacksmith ... For the sign jobs, add a short route-pick step that reads which Blacksmith macOS label has a live runner, prefer it, and fall back to 6vcpu. Then a stale variable can't strand the nightly again."

How the label is picked. scripts/ci/nightly_sign_runner.py reads the queue janitor's pool snapshot (macos-pool-load, at most 15 minutes old; two GITHUB_TOKEN reads per nightly run). It names the first candidate whose queue is moving:

  • the variable's label, if it is a Blacksmith macOS 26 label;
  • then blacksmith-6vcpu-macos-26;
  • then blacksmith-12vcpu-macos-26.

A label counts as not moving when the snapshot saw a job wait on it for 25 minutes or more: next_macos_cascade.STALL_MINUTES, the same rule cmux-next uses. With no fresh snapshot, the variable keeps its say, as today. The script never names an empty label and never fails its job.

Where it is read:

  • A new pick-nightly-sign-runner job (Linux, actions: read, continue-on-error) serves build-sign-notarize-nightly, verify-signed-nightly, verify-final-dmg-nightly and generate-nightly-deltas.
  • A step in find-nightly-next-recovery serves the recovery and verify-recovered jobs.
  • A step in legacy-identity-target serves sign, verify and publish-nightly-identity-legacy.
  • Each job's runs-on is picked || vars.MACOS_RUNNER_26 || 'blacksmith-6vcpu-macos-26', so a failed or skipped pick falls back to today's route.

Not changed:

  • No environment, secret, permission or trigger change. Signing jobs keep their environments.
  • Owned Macs still only build; build-nightly-app and prepare-nightly-variant-primary keep their owned-pool routes.

Tests:

  • tests/test_nightly_sign_runner.py (new; ci-guards release-notary, linux-guard) covers the pick rules and the wiring of every signing job. It fails if any other macOS job in nightly.yml routes on the variable alone.
  • tests/test_nightly_universal_build.sh is updated for the new runs-on and needs.
  • These pass locally: test_ci_nightly_off_blacksmith, test_nightly_trusted_routing, test_nightly_delivery_concurrency, test_ci_signing_secrets_need_release_environment, test_nightly_unsigned_resume, test_ci_test_execution_registry and test_ci_tui_routes_avoid_nightly_label.
  • test_ci_workflow_permissions_are_job_scoped and test_ci_self_hosted_guard.sh fail the same way on base (chief-evals.yml), so those reds are not from this PR.

Overlap: cmux#25404 (same lane) changes nightly.yml's build and legacy-sign concurrency groups. The two PRs touch different lines.

Risk: normal. Only where the signing jobs queue changes.

Agent session: agent session

šŸ¤– Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.