#27022 ci: the overflow watch and nightly control jobs never wait on Blacksmith Linux

On 2026-10-28, from about 20:00Z to 21:15Z, Blacksmith stopped starting jobs on every Linux size (2, 4, 8, 16 and 32 vCPU) across the org. Blacksmith macOS kept starting jobs within seconds, and status.blacksmith.sh reported all systems operational. Two parts of our own CI made the outage worse:

• The overflow switch couldn't decide. Run 40758329180 queued its probe (4 vCPU Linux) and its watch (16 vCPU Linux) at 20:07:53Z. The watch was waiting on the same pool the probe tests. That run held the workflow's concurrency group, so the scheduled runs at 20:09, 20:20, 20:34, 20:43 and 20:52 were replaced while pending. Nothing was decided until Blacksmith came back at 21:15Z.
• The nightly-next tick and the feed watch had nowhere else to run. Both are pinned to blacksmith-2vcpu-ubuntu-2404.

Changes:

• Overflow watch. It now always runs on blacksmith-6vcpu-macos-26, or on CI_CONTROL_RUNNER when that is set. Forks get macos-26. It already ran on that label while overflow was off: run 40755571607 today at 19:41Z. The probe still tests Blacksmith Linux, so the watch has to sit on another pool to judge it. The old 16 vCPU branch is removed.
• Concurrency group. The overflow workflow keeps its single fixed group. The switch's variable writes and its record delete are separate calls, so two watches must never run at once. Once a watch starts, it decides within CI_CLOUD_PROBE_MINUTES or force-cancels its own run, so it can't hold the group for long.
• New CI_CONTROL_RUNNER variable. It is unset by default, so nothing changes until someone sets it. When set, it moves the tick's request job and the feed watch's watch job off Blacksmith Linux. To support a macOS value, the tick's gh cache shim hashes with shasum -a 256 where sha256sum is missing. ci-health-report and ci-repo-variables now report the variable, and the live runs-on check reads it.
• promote-nightly-next is unchanged. It holds the release App credentials, and only a literal ephemeral label may carry them.
• Docs. docs/ci-runners.md covers all of the above. It also states what this does not fix: a nightly's Linux cross-builds still wait for Blacksmith Linux, or for the LINUX_RUNNER break-glass.

Risk: low. With the variable unset, the only behavior change is that the overflow watch takes a 6 vCPU macOS slot where it used to take a 16 vCPU Linux one, for about a minute every 10 minutes. The tick and the feed watch have not yet run on macOS. The docs say to check the first run after setting the variable.

Tests, all passing locally:

• test_ci_cloud_overflow_switch: watch runs-on and the fixed group
• test_ci_nightly_off_blacksmith: the CI_CONTROL_RUNNER branch and the portable hash
• test_ci_runs_on_values
• test_runner_label_policy
• test_ci_fork_runner_routing
• test_ci_signing_keychain_is_deleted_before_bundle_code
• test_ci_repo_variable_defaults
• test_nightly_delivery_concurrency
• test_nightly_promotion_admission
• test_ci_actionlint_covers_every_workflow
• test_ci_guard_workflow_structure
• test_ci_quota_hold_gate
• test_ci_self_hosted_guard.sh
• actionlint on every changed workflow

🤖 Generated with Claude Code

Agent session: agent session

Summary by CodeRabbit

• Chores
  • CI runner selection can be configured for overflow monitoring and nightly checks; existing fallback runners remain available when no override is set.
  • Nightly checks can generate SHA-256 cache keys using either of the common checksum utilities.
  • Overflow monitoring continues to use a fixed concurrency group, with overlapping runs prevented.
• Documentation
  • Updated CI runner guidance to describe runner overrides, fallback behavior, and overflow-monitoring details.