ci(nightly): Linux jobs take the runner set, not one small Blacksmith label

TE
8d ago
+117-73

Every Linux job in nightly.yml named one small Blacksmith size (4 or 8 vCPU), either as the vars.LINUX_RUNNER fallback or as the job's only label. When that one label stops starting runners, the nightly waits on it with no fallback. On 2026-10-28 signer-dmg, the last job of nightly-next run 40778774768 (25 of 26 jobs done), sat queued on blacksmith-4vcpu-ubuntu-2404 while the 16 and 32 vCPU labels kept serving, and the overflow switch's own probe on that label saw no runner for 5 minutes. The feed was about 4 hours behind.

Leo, 2026-10-28: "I hate pins. Like, just pin the set of the shit that isn't hilariously slow. Like most of what GitHub has is hilariously slow. We can't be using them."

What changes

Linux jobs now use the declared runner set (Blacksmith 32 and 16 vCPU, .github/runners.json):

  • Jobs that read vars.LINUX_RUNNER fall back to blacksmith-32vcpu-ubuntu-2404, not 4 or 8 vCPU. That covers 23 jobs, plus the unsigned-resume Linux branch of build-nightly-app and build-nightly-ghostty-cli-helper.
  • signer-build picks only from the set: 16 vCPU when LINUX_RUNNER names 16, otherwise 32. It no longer reads CI_CLOUD_OVERFLOW_SAVED.
  • prepare-nightly-variant-fallback always runs, because a skipped fallback skips signing and publishing. After a primary success it is a no-op on the primary's label (vars.LINUX_RUNNER, else 32). After a failed primary it retries on the other member of the set: 32 when LINUX_RUNNER names 16, otherwise 16. One stalled size can't hold both attempts.
  • The environment-bound signing and publishing jobs (signer-app, signer-dmg, keep-published-dsyms, publish-nightly-identity-legacy) keep a literal label, because test_ci_signing_keychain_is_deleted_before_bundle_code.py refuses a secret-holding job whose runner a variable picks. Their literal is 32 vCPU now.

Why the set-only jobs ignore a hosted LINUX_RUNNER

signer-build and the prepare retry never take hosted ubuntu-*, even if the overflow switch writes it into LINUX_RUNNER (the switch's hosted-Ubuntu failover, scripts/ci/cloud_overflow_switch.py). Hosted runners are outside the declared set under the rule quoted above, and a nightly signing path should never queue there. The switch writing a hosted label is being fixed in a separate PR. The other jobs that read vars.LINUX_RUNNER keep their existing override behavior here. The repo-wide pin sweep with its lint is a separate PR, so this one stays small.

Not changed: environments, permissions, secrets, pull_request_target, and every macOS route.

Tests

  • Expectations that pinned 4 or 8 vCPU now expect the set: test_ci_nightly_off_blacksmith.py, test_nightly_legacy_publish_linux.py, test_nightly_trusted_routing.py, test_nightly_unsigned_resume.py and tests/test_nightly_universal_build.sh.
  • New: test_prepare_fallback_follows_the_primary_then_retries_on_another_size checks that, for LINUX_RUNNER in {"", ubuntu-24.04, 4 vCPU, 16, 32}, the fallback follows the primary after a success, retries on another size after a failure, and that both the retry and signer-build land in {16, 32}.
  • Every test that reads nightly.yml, runs_on_labels, runners.json or LINUX_RUNNER passes locally, apart from two that fail the same way on base for host-tool reasons: test_nightly_next_arm64_only.py (no lipo on Linux) and test_check_ghostty_zig_workflows.py (no bashlex). nightly.yml is 489950 bytes, under the 490000 size test.

Risk: low. Linux jobs move to bigger sizes of the same Blacksmith provider, at the same trust level.

🤖 Generated with Claude Code

Agent session: agent session