ci(overflow watch): take the trusted CI_TRUSTED_RUNNER selector, not the dead macOS pin
The overflow watch (ci-cloud-overflow-probe.yml watch) named one literal label, blacksmith-6vcpu-macos-26, with no fallback. On 2026-11-03 RUNNER_POOLS marked that label dead (_dead since 17:34Z), and run 41688532704's watch queued from 17:53:18Z to 18:12:32Z (19 min). The watch is the only place the hosted light rescue (hosted_light_rescue.py, the 3-minute failover for the declared hosted light jobs) runs, and the workflow has one fixed concurrency group, so the rescue ran about every 20-25 min instead of every pass. Hosted light jobs waited out the whole hosted queue (p50 529 s, p90 621 s, max 717 s at 18:15Z). Tracked in manaflow-ai/cmuxterm-hq#6671.
Change. The watch gets its own bounded selector over CI_TRUSTED_RUNNER:
contains(fromJSON('["blacksmith-2vcpu-ubuntu-2404","blacksmith-8vcpu-ubuntu-2404"]'), vars.CI_TRUSTED_RUNNER) && vars.CI_TRUSTED_RUNNER || 'blacksmith-2vcpu-ubuntu-2404'
- It can only resolve to Blacksmith Linux 2 or 8 vCPU. Neither is a probe size (
PROBE_LABELS, 4 and 16 vCPU), so the watch never queues behind the probes it waits on, which was the 2026-10-28 failure. - Any other value takes 2 vCPU: unset (the switch's
plan_offcan unset it), 4 vCPU (ALTERNATE_FAILOVERS), a probe size, hosted, or a stray value. - It needs no macOS: the job is Python and gh only. The switch docstring now says Linux.
Trust is unchanged. Blacksmith runners are trusted like GitHub-hosted runners. The App key moves from one ephemeral Blacksmith VM class (macOS 6 vCPU) to two ephemeral Blacksmith Linux labels. The same GLAEDA_ROUTE_APP_KEY already runs on Blacksmith Linux through CI_TRUSTED_RUNNER in ci-owned-pool-rescue.yml. The expression bounds the variable, so no value reaches a reused or owned machine, and runner_label_errors passes it as a Blacksmith-only pick. This changes no runner trust list, secret, environment or workflow permission.
Tests.
tests/test_ci_cloud_overflow_switch.py: the watch must take exactly this selector, with itsephemeral-requirednote. It is evaluated for everyCI_TRUSTED_RUNNERvalue: unset, the adversarial values fromtest_ci_fork_runner_routing, the probe labels, hosted, the failover sizes and 2/8/32 vCPU. Each must resolve to 2 or 8 vCPU and never to a probe label. A mutation that sets the fallback to 4 vCPU fails it.tests/test_ci_runner_set.py: the watch is no longer in the pinned list.tests/test_ci_macos_xcode_selection.py: the watch's no-Xcode macOS exemption is removed.- Run locally and passing:
test_ci_cloud_overflow_switch,test_ci_runner_set,test_ci_macos_xcode_selection,test_ci_runner_pools,test_ci_fork_runner_routing,test_ci_repo_variable_defaults,test_ci_runs_on_values,test_ci_no_github_hosted_macosandtests/test_ci_self_hosted_guard.sh.
Follow-ups (#34232).
- The watch can no longer reach GitHub-hosted Ubuntu, even when
CI_TRUSTED_RUNNERis set to it by hand; it takes 2 vCPU instead. docs/ci-runners.mdand the watch's.github/warm-sources.jsonentry still describe macOS. They are kept out of this PR so that it changes only CI code.
Not in this PR: fleet-first light jobs. The fleet Linux label cmux-fleet-linux-x64 has no runner to offer right now. The GCP runner reconciler reports credential_provider_broken (workload identity invalid_target) since 2026-11-01T11:57Z, with 0 available and 0 booting. The rule allows hosted Ubuntu only after a live fleet runner and never by returning an unserved fleet label, so with no live fleet runner the light jobs' order today is hosted Ubuntu, then Blacksmith on the re-run. That is what they already do, once this PR makes the failover fire. The routing change follows in its own PR once the grant is in place (cmuxterm-hq#6671).
Performance
| metric | before | after | delta | harness or run link |
|---|---|---|---|---|
| watch queue on its runner label | 19 min (run 41688532704, dead blacksmith-6vcpu-macos-26) | measured after merge | gh run list --workflow ci-cloud-overflow-probe.yml | |
| watch pass wall time, queue included, 40 runs 13:17Z-19:43Z | p50 8.5 min, max 40.6 min | measured after merge | same | |
| hosted light rescue cadence | about 20-25 min during the dead-label stretch | one pass per watch run (*/10 cron) | cmuxterm-hq#6671 | |
hosted ubuntu-24.04 queue wait, cmux PR jobs | p50 529 s, p90 621 s (18:15Z sample) | measured after merge | cmuxterm-hq#6671 | |
| Blacksmith minutes per day, the watch | about 150 runs/day on macOS 6 vCPU | the same runs on Linux 2 vCPU (est.); minute count unchanged, cheaper class | macOS minutes to Linux minutes | runner labels |
Agent session: agent session
š¤ Generated with Claude Code