ci(overflow): main's switch never points a signing runner variable at an owned mini

Merged
TE
9d ago
+206-11

Why

The overflow switch (ci-cloud-overflow-probe.yml) runs on a schedule, so it runs from main. On 2026-10-28 it failed over three times (21:59Z, 22:13Z, 22:53Z) and each time pointed MACOS_RUNNER_26 at the std owned pool. Signing jobs on feat-cmux-next read that variable in runs-on (rule 7 of tests/test_ci_signing_keychain_is_deleted_before_bundle_code.py), so cmux-next-guards.yml's signing runner variables are ephemeral job went red on feat-cmux-next and on every pull request into it. The variables are repository-wide.

#27094 fixes this in the switch, but on feat-cmux-next. The scheduled run never executes that copy, so on its own it cannot stop the re-fire. This PR ports #27094 onto main's newer switch (Linux failover to another Blacksmith size, the hosted-Ubuntu record migration). Keep both: #27094 also adds the rule 7 test's list check, which exists only on feat-cmux-next.

What

  • scripts/ci/signing_runner_variables.py: the one list of variables signing jobs read (LINUX_RUNNER, MACOS_RUNNER_26) and the ephemeral rule (blacksmith-* or ubuntu-*). It matches #27094's file plus one note that main carries a copy.
  • cloud_overflow_switch.py:
    • MACOS_RUNNER_26 is removed from STD_VARIABLES. While Blacksmith is down, its jobs wait for Blacksmith macOS rather than moving to a reused mini.
    • MACOS_RUNNER_26 is also removed from what CI_CLOUD_FAILOVER may name. On main, every value an override could give it is already refused: GitHub-hosted, a Blacksmith label for a macOS variable, or a reused machine.
    • An override that points LINUX_RUNNER at a non-ephemeral label is refused.
    • plan_off and plan_resume never write a non-ephemeral label to a listed variable. That covers records written before this change, like today's, which still name MACOS_RUNNER_26 with an owned after. plan_on still restores a recorded value.
  • check_repo_variables.py never excuses a listed variable that names a reused machine, even when a switch record names it.
  • The guard is unchanged and not loosened. Owned-Mac signing keeps its own fail-closed route (#26944, #27066, #27113): a picker output in runs-on, never a shared overflow variable. If signing jobs later choose their runner through that picker alone, the guard can check the pick instead. That change belongs with those PRs.

Not in this PR (coordination)

  • The switch already restores itself. When its probe starts again it puts back the record's before values and deletes the record; it did so at 23:21:20Z today, and the guard went green after that.
  • Probing more than one label and stall-aware failover are separate work in flight. #27022 moves the watch off Blacksmith Linux. This PR leaves the probe and the workflow alone, so it does not conflict with either.

Tests

  • tests/test_ci_cloud_overflow_switch.py: 46 pass. The signing-variable tests are ported from #27094 and adapted: on main, no override can move MACOS_RUNNER_26, and a Blacksmith Linux override for LINUX_RUNNER is kept.
  • tests/test_ci_check_repo_variables.py and tests/test_ci_no_github_hosted_macos.py pass.

Risk: low. CI routing only. While Blacksmith macOS is down, jobs that read MACOS_RUNNER_26 queue for it instead of moving to the owned std pool; rule 7 requires that. The other macOS variables still fail over to the owned pools as before.

🤖 Generated with Claude Code

Agent session: agent session

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by cubic

Stops main's scheduled overflow switch from ever pointing signing runner variables at reused machines, fixing the intermittent signing runner variables are ephemeral guard failures on feat-cmux-next and its pull requests.

The switch runs from main, so the existing feat-cmux-next fix cannot stop it: on 2026-10-28 its failover pointed MACOS_RUNNER_26 at the owned std pool three times, turning the guard red on every PR.

Changes

  • Adds scripts/ci/signing_runner_variables.py as the single list of variables signing jobs read (LINUX_RUNNER, MACOS_RUNNER_26) and the ephemeral rule (blacksmith-* or ubuntu-*).
  • Removes MACOS_RUNNER_26 from the owned failover and from what overrides may name; its jobs wait while Blacksmith is down.
  • Refuses overrides naming a reused machine for a signing variable, and a stalled probe never re-applies an older record's owned value; restoring overflow still puts back the recorded value.
  • check_repo_variables.py never excuses a reused machine in these variables, even when a switch record names it.

Risk is low: CI routing only. The other macOS variables still fail over to the owned pools, and owned-Mac signing keeps its fail-closed picker route. Keep the signing_runner_variables.py list in sync with the feat-cmux-next copy.

Summary by CodeRabbit

  • Bug Fixes
    • Signing jobs are protected from failover settings that would route restricted runners to reused machines.
    • Unsafe runner overrides and retry destinations are ignored, while valid failover restoration continues to work.
    • Runner-variable checks no longer treat a saved overflow record as sufficient to allow an invalid restricted runner value.
  • Chores
    • CI checks now run when signing-runner configuration changes.