ci(overflow): probe the 32 and 16 vCPU Linux sizes; trip only when none starts

Merged
TE
8d ago
+478-169

Why

The overflow switch probed a single label: the record's, else LINUX_RUNNER, else blacksmith-4vcpu-ubuntu-2404. On 2026-10-28 the 4 vCPU label stalled on its own at 21:59Z, 22:13Z, 22:53Z and 23:30Z while the other Blacksmith Linux sizes kept starting jobs. Each of those stalls flipped every overflow variable, including the macOS ones. A single probed label is a pin.

What

  • PROBE_LABELS = ("blacksmith-32vcpu-ubuntu-2404", "blacksmith-16vcpu-ubuntu-2404"), the sizes compiles run on. The probe job is now a matrix with one job per label, named Blacksmith probe (<label>). A drill (probe_label) still probes only its own label and is always a dry run.
  • The probe set decides only after every probe has decided. Every probe stalled: overflow goes off. Every probe started: an existing record is put back and deleted. Mixed results (outcome=partial): nothing changes. A single stalled size never flips the fleet, and a record is not cleared while a size jobs fall back to is still stalled (fast-checks.yml takes 32 vCPU when the record is gone). This answers review finding 1.
  • Health is tracked per label. A Linux failover, a legacy hosted-Ubuntu migration, or a resume of a partial switch names only a label whose probe started in this run, and never GitHub-hosted Ubuntu. When no Linux size started, the Linux variables are left alone and the owned macOS failovers still apply. This answers review finding 2.
  • When some probes start and others stall, the switch changes nothing. It doesn't flip overflow, doesn't clear the record and doesn't move any variable; it only writes CI_CLOUD_LABEL_HEALTH. Moving work off one stalled size, including workflows that name a size literally (fast-checks.yml), is the RUNNER_POOLS reconciler's job (#27129), and it takes this health as its input. An earlier round of this PR moved Linux variables on partial results. That was dropped after review: a Linux-only record made a later full stall skip the macOS failover.
  • A restore puts a Linux variable back only onto a size whose probe started. If its value before is a label nobody probed (an older record's blacksmith-4vcpu-ubuntu-2404, a legacy CI_TRUSTED_RUNNER 4 vCPU, or unset), it stays where it is. A GitHub-hosted value before is never put back. This covers second review finding 1 and third review finding 2.
  • When every probe starts and a legacy record left a Linux variable on GitHub-hosted Ubuntu (before an unprobed 4 vCPU), the hosted value no longer outlives the record. LINUX_RUNNER moves to a probed size that started. CI_TRUSTED_RUNNER is unset, because its readers accept no probed size, so they fall back to Blacksmith (fourth review).
  • No code path writes a GitHub-hosted label to any variable, and a test checks it across stall, partial, start, override and legacy-record cases. HOSTED_LINUX is already gone from main. The feat-cmux-next copy of the switch still has it, but the scheduled switch runs from main.
  • When no Linux size started, stuck Blacksmith Linux jobs are no longer force-cancelled and re-run. A re-run would land on the same label, and the job would be cancelled again on every scheduled run. macOS jobs still move to the owned pools (second review, finding 2).
  • Each deciding run that has the switch token writes CI_CLOUD_LABEL_HEALTH as {"checked", "run", "minutes", "labels": {label: "started" | "stalled"}}. The switch never reads it back. It is there so the no-pins RUNNER_POOLS reconciler (runner_pools.py, which will replace this switch's flipping) can read per-label health in the same form.
  • The watch always runs on blacksmith-6vcpu-macos-26, never on a size the probes ask for. Unless every probe started, the run force-cancels itself, so a queued probe cannot hold the concurrency group.
  • The record no longer decides what is probed. Its probe field only names the set.
  • check_repo_variables.py now checks LINUX_RUNNER and MACOS_RUNNER_26 independently of the drift policy, which accepts some labels no signing job may run on. This addresses CodeRabbit's finding on #27118.

Overlap

  • #27022 also changes the watch's runs-on: it moves to the same blacksmith-6vcpu-macos-26, or CI_CONTROL_RUNNER. Whichever PR lands second takes the other's line. They don't conflict in intent.
  • #27118 (merged as 71708aa8) keeps MACOS_RUNNER_26 and LINUX_RUNNER off owned minis. This PR builds on it and doesn't change that.
  • Stall-aware failover and the no-pins lint are separate work. This PR changes only the probe and the switch.

Tests

  • tests/test_ci_cloud_overflow_switch.py: 56 pass. New tests cover:
    • a mixed result writes only per-label health: no restore, no flip, no record;
    • a legacy hosted-Ubuntu record never migrates or resumes onto a stalled size or onto hosted Ubuntu;
    • Linux failovers name only a size that started;
    • a restore never moves a Linux variable onto an unprobed label;
    • no Linux rescue happens when no Linux size started;
    • the matrix restates PROBE_LABELS;
    • the failover never names a stalled size;
    • a signing variable the drift policy accepts is still reported. This test fails against the old checker.
  • tests/test_ci_check_repo_variables.py, test_ci_fast_checks_failover.py, test_ci_no_github_hosted_macos.py, test_ci_runs_on_values.py, test_ci_workflow_concurrency.py, test_ci_fork_runner_routing.py, test_ci_macos_xcode_selection.py and actionlint pass.

Risk: low. CI routing only. When one size stalls alone, jobs on that size queue for it instead of the whole fleet failing over. One new repository variable (CI_CLOUD_LABEL_HEALTH) is written with the token the switch already uses.

🤖 Generated with Claude Code

Agent session: agent session

Summary by CodeRabbit

  • CI Improvements
    • Overflow checks now test both configured Linux runner sizes independently, while monitoring continues on a fixed macOS runner.
    • Linux routing changes only to runner sizes confirmed to have started. Mixed probe results leave routing and saved recovery state unchanged.
    • Runner health results are recorded per size when a decision is made; drills and dry runs do not record health.
    • Repository checks now report signing-runner values that fall outside the allowed label.
  • Documentation
    • Updated the CI runner guide to explain probe outcomes, routing behavior, and recovery conditions.