ci(overflow): never point a signing runner variable at an owned mini

Merged
TE
9d ago
+207-17

Why

On 2026-10-28 at 21:59Z the cloud overflow switch (ci-cloud-overflow-probe.yml, run 40775549143) failed over and pointed every STD_VARIABLES entry at the std owned pool, MACOS_RUNNER_26 included. Signing jobs (for example auto-resume-nightly-notarization.yml) read MACOS_RUNNER_26 in runs-on, so cmux-next-guards.yml's signing-runner-variables job went red on every feat-cmux-next pull request (rule 7 of tests/test_ci_signing_keychain_is_deleted_before_bundle_code.py: a job holding a signing secret runs on an ephemeral runner; Leo, 2026-10-24: "blacksmith runners we can simply trust"). The variable was put back by hand at 22:05Z, and at 22:13Z the next stalled probe (run 40777608708) re-applied it through plan_resume, because the restored value equalled the record's before.

What

  • scripts/ci/signing_runner_variables.py is the one list of variables signing jobs read (LINUX_RUNNER, MACOS_RUNNER_26) and the ephemeral rule (blacksmith-* or ubuntu-*). The switch, check_repo_variables.py and the rule 7 test all read it, and the rule 7 test now fails if the signing-runner-variables guard job checks anything other than exactly that list.
  • cloud_overflow_switch.py:
    • MACOS_RUNNER_26 leaves the owned failover (STD_VARIABLES). Its jobs wait for Blacksmith instead of landing on a reused machine.
    • CI_CLOUD_FAILOVER may still name it, but an override that names a non-ephemeral label for a listed variable is refused with a warning.
    • plan_off and plan_resume never write a non-ephemeral label to a listed variable, even when an older record names one. plan_on still restores a recorded value.
  • check_repo_variables.py never excuses a listed variable at a non-ephemeral label just because a record names it.
  • The guard is unchanged and not loosened. Owned-Mac signing keeps its own routing (cmux#26944, cmux#27066), not a shared-pool variable.

Before / after

casebeforeafter
fresh stall, MACOS_RUNNER_26=blacksmith-6vcpu-macos-26set to glaeda-std-xcode-26.6, guard redleft alone
stall with a record naming MACOS_RUNNER_26 at its beforere-applied the owned failoverskipped, named in the summary
overflow back on with such a recordrestores beforerestores before (unchanged)
guard job loops over a different list than rule 7not detectedrule 7 test fails

Tests

  • tests/test_ci_cloud_overflow_switch.py: 38 pass (9 new). Run against the old switch, 5 of the new switch tests fail, and against the old check_repo_variables.py the new checker test fails.
  • tests/test_ci_signing_keychain_is_deleted_before_bundle_code.py passes, and fails when the guard's loop drops MACOS_RUNNER_26.
  • tests/test_ci_check_repo_variables.py, tests/test_ci_blacksmith_linux_failover.py, tests/test_ci_linux_compile_runners.py, tests/test_ci_fork_runner_routing.py, tests/test_cmux_next_guardrails.py, tests/test_ci_guard_attribution.py pass.

Risk: low. CI routing only; while Blacksmith macOS is down, jobs on MACOS_RUNNER_26 queue for it instead of moving to the owned pool, which is what rule 7 requires.

Agent session: agent session

šŸ¤– Generated with Claude Code