ci(nightly): sign and verify picks stay on blacksmith-6vcpu-macos-26, never the stalled 12 vCPU label
Summary
Main's classic nightly can sign only on Blacksmith, and its sign picker can send the signing jobs to blacksmith-12vcpu-macos-26. That label hasn't started a job since 2026-10-28 20:32Z. In the last 100 nightly.yml runs (back to 2026-10-29 06:18Z), 0 of the 10 jobs that asked for it got a runner. Main's run 40834485210 shows the result. Its re-run (attempt 2) took the picker's 12 vCPU alternate, and build-sign-notarize-nightly (arm64, x86_64 and universal) has waited on it since 07:47Z.
pick-classic-sign-runner (via classic_nightly_build_runner.py) could reach the 12 vCPU label in two ways:
alternate, which every re-run of failed jobs takes, is always "the other label". When the pick is 6 vCPU, the alternate is 12 vCPU.- When the queue snapshot sees 6 vCPU stalled, the first pick itself is 12 vCPU. The snapshot can't see that a label has no capacity while nothing waits on it, so an empty 12 vCPU queue looks like it moves.
This PR adds ONLY (a candidate label or its size) to the picker. When ONLY is set, that label is both cloud and alternate, whatever the snapshot says. pick-classic-sign-runner and the runner pick in cmux-next-dev-daily.yml set ONLY: 6vcpu. The DEV daily has the same exposure: its verify, tip-dmg-build and tip-deltas jobs read the same cloud/alternate outputs (when no owned Mac takes them). The runs-on expressions don't change, and the labels stay Blacksmith-only literals. Rule 6/7 of the signing guard and its PICKED_EPHEMERAL_VERIFY_JOBS entry still match. Owned Macs stay out of signing.
Follow-up of #27440, which moves release.yml's hard-coded build-sign-notarize off the same label.
Trade-off: these jobs can no longer move to 12 vCPU while 6 vCPU is slow. When the owned-pool rescue moves a stalled sign job, the re-run waits on 6 vCPU again and loses its place in the queue. On main the rescue acts on attempt 1 only (signs_late), so this happens at most once per run. Before this change, that re-run went to a label that started nothing.
Testing
Regression: python3 tests/test_classic_nightly_build_runner.py. The new SignPicksStayOnSixVcpu reads each picker step's env from the two workflows and runs main() for 4 cases:
- 6 vCPU moving
- 6 vCPU stalled while 12 vCPU looks idle
- both labels stalled
- no snapshot
Each case runs on attempts 1 and 2. On 4eed88bab78 (test only) it fails 16 subtests: the pick or the alternate is 12 vCPU. On the head, all 23 tests pass.
Observed / Not run
Observed, run locally on the head, all pass:
- The picker test above.
test_ci_owned_pool_rescue.py(336 tests).test_ci_signing_keychain_is_deleted_before_bundle_code.py.test_ci_publisher_mac_jobs_move.py,test_ci_pr_runner_pool.pyandtest_ci_queue_janitor.py.test_ci_quota_hold_gate.py,test_cmux_next_dev_daily_dispatch.pyandtest_cmux_next_dev_tip_feed.py.test_delivery_workflows_cancel.py,test_dev_app_linux.pyandtest_glaeda_route_app_budget.py.test_nightly_delivery_concurrency.py,test_publish_app_token_workflow.py,test_ci_runs_on_values.pyandtest_runner_label_policy.py.test_ci_no_github_hosted_macos.py,test_ci_workflow_concurrency.pyandtests/test_nightly_universal_build.sh.- actionlint 1.7.7 (
SHELLCHECK_OPTS=-S warning) on both workflows: clean.
The new workflow comments name the 12 vCPU label only in words. owned_pool_rescue.retry_labels() reads every Blacksmith label named in the workflow text, and naming the label literally broke test_the_dev_daily_retry_label_comes_from_its_workflow.
test_ci_owned_pool_sources.py fails 1 check, the same way on origin/main 18f2c43e955. That check reads the origin/feat-cmux-next tree (sign-nightly-identity-legacy-blacksmith not in NIGHTLY_SIGN_ALTERNATES), so this PR doesn't cause it.
Not run: a nightly or DEV daily run with this picker. The DEV daily checks out the picker from the default branch, so ONLY takes effect there only after merge. The stuck run 40834485210 isn't touched; it still needs a cancel and a full re-run, or the next scheduled nightly.
Changelog
none