ci(overflow): the watch that holds the variable-write key keeps a literal ephemeral label

Closed
LA
8d ago
+47-11

Summary

Since #27022 (thanks @teamleaderleo for the Blacksmith Linux outage fix), the overflow watch job in ci-cloud-overflow-probe.yml takes its runner from vars.CI_CONTROL_RUNNER, and its comment suggests an owned Mac label. The same job mints the GLAEDA_ROUTE_APP_KEY token, which can write repository variables. As a result:

  • a repository variable could put the variable-write key and its token on a reused self-hosted Mac, where they can outlive the job (owned Macs also run same-repository PR code);
  • the job could write the variable that picks its own runner.

This PR puts the watch back on the literal blacksmith-6vcpu-macos-26. promote-nightly-next keeps its literal label for the same reason (#27022: "only a literal ephemeral label may carry" App credentials). nightly-next-tick.yml and nightly-next-feed-watch.yml keep CI_CONTROL_RUNNER. I checked them: they hold no secret and no App token, only their job-scoped GITHUB_TOKEN.

Why not split the watch. One option is a read-only watch on the movable runner plus a separate writer that holds the key on a literal label. That split gives no benefit here. The repo allows only Blacksmith as a literal ephemeral label, because check_no_github_hosted_runners (Leo, 2026-10-27) refuses GitHub-hosted labels. So the writer would wait on Blacksmith macOS, the same as this watch does. A first version of this branch put the writer on ubuntu-24.04, and that guard refused it.

What stays of Leo's outage path:

  • The 2026-10-28 case that #27022 fixed still works: when Blacksmith Linux stops and Blacksmith macOS serves, the watch runs on macOS and switches overflow.
  • CI_CONTROL_RUNNER still moves the nightly control jobs.
  • When all of Blacksmith stops, the watch waits, and an operator points the variables at their failovers by hand, as on 2026-09-29. No ephemeral runner that the repo allows serves then, so no automatic path can hold the key.

New guards in tests/test_ci_cloud_overflow_switch.py:

  • every job of the overflow workflow that holds a secret runs on a literal blacksmith-* label;
  • no job in any workflow that CI_CONTROL_RUNNER moves holds a secret or mints an App token.

Testing

  • python3 tests/test_ci_cloud_overflow_switch.py: 3 failures on the test commit 57e78404761 (the base workflow), and 62 passes on the head.
  • Also pass on the head: tests/test_ci_runner_pools.py, tests/test_ci_nightly_off_blacksmith.py, tests/test_ci_fork_runner_routing.py, tests/test_ci_repo_variable_defaults.py, tests/test_ci_macos_xcode_selection.py, tests/test_ci_fast_checks_failover.py, tests/test_ci_change_areas.py, tests/test_ci_signing_keychain_is_deleted_before_bundle_code.py, bash tests/test_ci_self_hosted_guard.sh, and python3 scripts/verify-local.py --affected origin/main (16/16).
  • actionlint 1.7.7 (the pinned version, with shellcheck): clean on ci-cloud-overflow-probe.yml.

Observed / Not run

  • Observed: the commands above, on a Linux VM with portable Python only.
  • Not run: a live run of the workflow. The next scheduled run on main is the first live check that the watch starts on blacksmith-6vcpu-macos-26. That label is the one it used before #27022, and it is also the default when CI_CONTROL_RUNNER is unset.

Changelog

none

Proof

CI only: no user-visible change.

Checklist

  • Behavior changes have added or updated tests, or Testing says why not
  • Reviewed with a subagent before merge (cmux-review), and all bot and human review comments resolved

Summary by cubic

Pins the overflow watch back to the literal blacksmith-6vcpu-macos-26 label so the repository-variable write key (GLAEDA_ROUTE_APP_KEY), which the job mints, can't land on a reused machine or write the variable that picks its own runner. Since #27022 the watch took its runner from CI_CONTROL_RUNNER. The nightly tick and feed watch keep CI_CONTROL_RUNNER: they hold no secret. The 2026-10-28 Blacksmith Linux failover path from #27022 still works; when all of Blacksmith stops, the watch waits and variables are set by hand.

Adds guards that every overflow job holding a secret runs on a literal blacksmith-* label, and that no job CI_CONTROL_RUNNER moves holds a secret.

Summary by CodeRabbit

  • CI
    • Overflow monitoring now runs on a designated macOS runner and waits if that runner is unavailable. Nightly control jobs remain configurable.
  • Documentation
    • Updated CI runner guidance to clarify which jobs use a designated runner and which can use configurable runners.