ci(nightly): route release work through pooled capacity
Changelog
Changed: nightly release work runs on pooled capacity instead of a reserved host.
Main cache-writing builds take the trusted CI_SEED_TRUSTED_POOL; NEXT initial push and schedule builds take the controller-managed CI_SIDE_LANE_RUNNER pool that dev builds use. Retries, dispatches and fast builds fall back to the trusted Blacksmith macOS pool. Linux control, promotion, publication and receipt jobs use the Linux pool. The retired single-host nightly tuple is gone.
This addresses Leo's request:
"if people can, if we can build dev builds on any machine, we should also be able to do nightlys on any machine."
The nine jobs this deliberately does not reroute
Every job that holds signing, notarization or release-publishing material keeps the literal runner label it had on the base — macos-26 or ubuntu-24.04 — and each carries a comment saying why:
promote-nightly-next, promote-nightly-next-acked, build-sign-notarize-nightly, recover-nightly-next-notarization, publish-nightly, sign-nightly-identity-legacy, publish-nightly-identity-legacy, generate-nightly-deltas, sign-release-notes.
Rule 7 of tests/test_ci_signing_keychain_is_deleted_before_bundle_code.py forbids a secret-bearing job from selecting its runner through a repository variable, and gives the reason: a variable can be repointed onto a reused machine, where keychain material outlives the job. An earlier revision of this PR rerouted all nine onto vars.MACOS_RUNNER_26 || 'blacksmith-…' and vars.LINUX_RUNNER || …, which took that guard from OK: 18 keychain jobs delete the signing keychain before any bundle code runs on the merge base to nine failures. It also deleted the in-file comment warning against it. Both are restored.
This keeps the PR clear of runner-trust policy, which is not a routing decision. Leo's 2026-10-24 order ("blacksmith runners we can simply trust … build, sign, notarize and publish may run on one Blacksmith runner in the release environment") does allow signing on Blacksmith — via a literal Blacksmith label in the release environment, which satisfies rule 7 too. Moving these nine there is a separate change with its own review, not a side effect of pooling capacity.
Keys remain in protected GitHub environments; no signing secret is written to a runner or artifact. Forks retain their hosted image route.
The warm build slot
build-nightly-app keeps cancel-in-progress: false, and its concurrency group now gates on the same vars.CI_PR_POOL_OWNED == '1' that its runs-on does. Before that fix the group gated only on vars.CI_SIDE_LANE_RUNNER != '', so with owned capacity off and that variable set, Blacksmith fallback builds joined nightly-next-build-app-warm: a second arrival queues behind the holder and a third cancels the one pending. That contradicted both the comment above the block ("every other route … keeps a slot per run") and nightly freshness. Only the owned warm route shares the slot now; every other route gets a slot per run.
Guards, and a correction to how they were checked
Two guards that this PR had edited were demanding the unsafe form, so they are fixed as part of it:
test_ci_nightly_off_blacksmith.py:test_secret_bearing_nightlies_use_the_trusted_owned_poolasserted these jobs must readvars.MACOS_RUNNER_26. It is nowtest_secret_bearing_nightlies_pin_a_literal_runner, asserting the invariant instead. The file'stest_no_nightly_job_reads_a_shared_blacksmith_runner_variable, which this PR had deleted wholesale, is restored astest_no_secret_bearing_job_reads_a_shared_runner_variable— scoped to the secret-bearing jobs, which is where it still holds now that the blanket "no nightly on Blacksmith" rule is retired.test_nightly_universal_build.sh: same expectation forsign-nightly-identity-legacy, back to the literal. Itsverify-nightly-identity-legacyblock holds no environment and no secret, so that one keeps the pooled selector.
The earlier validation line in this description was wrong, and that is why the guard failure reached review. It reported python3 -m unittest … tests.test_ci_signing_keychain_is_deleted_before_bundle_code … (86 passed). That guard's checks run in its main(), not in unittest test methods, so -m unittest imported the module, found no failing case, and reported a pass while the guard itself exited 1. ci-guards.yml executes the file directly, which is the only way to run it.
Red, then green
Same commands, run by executing the files:
test_ci_signing_keychain_is_deleted_before_bundle_code.py | |
|---|---|
merge base 12e963d2 | OK: 18 keychain jobs delete the signing keychain before any bundle code runs, exit 0 |
blocked head 6da03b0c | 9 jobs fail, exit 1 |
| this head | OK: 18 keychain jobs …, exit 0 |
Mutation: putting vars.MACOS_RUNNER_26 || 'blacksmith-12vcpu-macos-26' back on build-sign-notarize-nightly is caught three times — the literal-runner assertion, the restored shared-variable rule, and the signing guard itself.
Observed / Not run
Observed, all by executing the files rather than importing them: the three signing-guard runs above; test_ci_nightly_off_blacksmith.py 12 tests OK; test_nightly_universal_build.sh pass; test_nightly_trusted_routing.py, test_runner_label_policy.py, test_nightly_unsigned_resume.py, test_ci_nightly_push_throttle.py, test_check_concurrency_generic_deinit.py, test_ci_tui_routes_avoid_nightly_label.py pass; .github/workflows/nightly.yml parses; actionlint adds no new class (its macos-26 label notes go from 12 on the base to 7 here, and the workflow_dispatch input-count note is on the base too).
Base findings, red the same way at the merge base and not from this PR: tests/test_ci_pr_runs_cancel_superseded.py (cmux-next-media.yml pull-request groups) and tests/test_ci_self_hosted_guard.sh (cmux-next.yml:3591 CI_COMPILE_CHECK_GATEWAY falls back to a GitHub-hosted label).
Not run: a real nightly. Pool selection, warm-slot behaviour under two concurrent pushes, and signing on the restored labels are all read from the workflow and its guards, not observed end to end. The nine signing jobs are on exactly the labels they used before this PR, so their behaviour is unchanged by construction; the routing change this PR does make is exercised the first time nightly-next pushes.
🤖 Generated with Claude Code
Agent session: agent session