ci(overflow): the watch that holds the variable-write key keeps a literal ephemeral label

Merged
LA
8d ago
+48-11

Summary

Replaces #27533, whose commits had an empty author email, which made CLA Assistant fail closed. The tree is the same, plus one doc sentence.

Since #27022 (thanks @teamleaderleo for the Blacksmith Linux outage fix), the overflow watch job in ci-cloud-overflow-probe.yml takes its runner from vars.CI_CONTROL_RUNNER, and its comment suggests an owned Mac label. The same job mints the GLAEDA_ROUTE_APP_KEY token, which can write repository variables. As a result:

  • a repository variable could put the variable-write key and its token on a reused self-hosted Mac, where they can outlive the job (owned Macs also run same-repository PR code);
  • the job could write the variable that picks its own runner.

This PR puts the watch back on the literal blacksmith-6vcpu-macos-26. promote-nightly-next keeps its literal label for the same reason (#27022: "only a literal ephemeral label may carry" App credentials). nightly-next-tick.yml and nightly-next-feed-watch.yml keep CI_CONTROL_RUNNER. I checked them: they hold no secret and no App token, only their job-scoped GITHUB_TOKEN.

Why not split the watch. One option is a read-only watch on the movable runner plus a separate writer that holds the key on a literal label. That split gives no benefit here. The repo allows only Blacksmith as a literal ephemeral label, because check_no_github_hosted_runners (Leo, 2026-10-27) refuses GitHub-hosted labels. So the writer would wait on Blacksmith macOS, the same as this watch does. A first version of this branch put the writer on ubuntu-24.04, and that guard refused it.

What stays of Leo's outage path:

  • The 2026-10-28 case that #27022 fixed still works: when Blacksmith Linux stops and Blacksmith macOS serves, the watch runs on macOS and switches overflow.
  • CI_CONTROL_RUNNER still moves the nightly control jobs.
  • When all of Blacksmith stops, the watch waits, and an operator points the variables at their failovers by hand, as on 2026-09-29. No ephemeral runner that the repo allows serves then, so no automatic path can hold the key.

New guards in tests/test_ci_cloud_overflow_switch.py:

  • every job of the overflow workflow that holds a secret runs on a literal blacksmith-* label;
  • no job in any workflow that CI_CONTROL_RUNNER moves holds a secret or mints an App token.

Testing

  • python3 tests/test_ci_cloud_overflow_switch.py: 3 failures on the test commit 57e78404761 (the base workflow), and 62 passes on the head.
  • Also pass on the head: tests/test_ci_runner_pools.py, tests/test_ci_nightly_off_blacksmith.py, tests/test_ci_fork_runner_routing.py, tests/test_ci_repo_variable_defaults.py, tests/test_ci_macos_xcode_selection.py, tests/test_ci_fast_checks_failover.py, tests/test_ci_change_areas.py, tests/test_ci_signing_keychain_is_deleted_before_bundle_code.py, bash tests/test_ci_self_hosted_guard.sh, and python3 scripts/verify-local.py --affected origin/main (16/16).
  • actionlint 1.7.7 (the pinned version, with shellcheck): clean on ci-cloud-overflow-probe.yml.

Observed / Not run

  • Observed: the commands above, on a Linux VM with portable Python only.
  • Observed by the review subagent from live runs today, with CI_CONTROL_RUNNER unset (so the label was the same as this PR's): watch jobs queued 6 to 39 minutes on blacksmith-6vcpu-macos-26 while both Linux probes started in seconds (run 40938852843: created 19:18Z, watch started 19:57Z). A waiting watch holds the fixed concurrency group, so later scheduled runs were cancelled (40940070200, 40941668804, 40942901017). This PR does not change that, because the variable is unset. It does remove the only way to move the watch without a PR. A follow-up could add a selector bounded to Blacksmith labels, like the CI_TRUSTED_RUNNER one; such a selector cannot name a reused machine.
  • Not changed here, and older than #27022: the same GLAEDA_ROUTE_APP_KEY is minted in jobs whose runner comes from vars.LINUX_RUNNER with no allowlist (for example ci-queue-janitor.yml, ci-hosted-queue-rescue.yml, test-e2e.yml). What limits that today: LINUX_RUNNER is a signing variable, so the switch never writes a reused machine to it, and check_repo_variables.py reports drift.
  • Not run: a live run of the workflow. The next scheduled run on main is the first live check that the watch starts on blacksmith-6vcpu-macos-26. That label is the one it used before #27022, and it is also the default when CI_CONTROL_RUNNER is unset.

Changelog

none

Proof

CI only: no user-visible change.

Checklist

  • Behavior changes have added or updated tests, or Testing says why not
  • Reviewed with a subagent before merge (cmux-review), and all bot and human review comments resolved

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by cubic

Restores the overflow watch job to the literal blacksmith-6vcpu-macos-26 runner label instead of using CI_CONTROL_RUNNER when set. The watch mints the GLAEDA_ROUTE_APP_KEY token that writes repository variables, so a variable-driven runner could place the key on a reused machine where it outlives the job and could rewrite its own runner.

Adds tests that require secret-holding jobs in the overflow workflow to use literal Blacksmith labels and that reject secrets in any job CI_CONTROL_RUNNER moves.

Updates docs/ci-runners.md to document the literal-label rule and the failover behavior when all Blacksmith instances are down.

Summary by CodeRabbit

  • Chores
    • The CI overflow watch now runs on a fixed Blacksmith runner. If Blacksmith capacity is unavailable, it waits for an operator to configure a failover.
    • The CI_CONTROL_RUNNER override is limited to eligible nightly jobs; credential-sensitive jobs remain on fixed cloud runners.
  • Documentation
    • Updated CI runner guidance to reflect runner selection and failover behavior.
  • Tests
    • Added checks for fixed runner assignments and restrictions on which jobs can use the override.