ci(nightly-next): canary and train compile take an owned Mac, then Blacksmith; nightly guards stop exempting hosted macOS

Merged
TE
10d ago
+138-50

Fixes the main half of HQ issue 4720. The ask: "GitHub-hosted macOS is never used, not even as a fallback, and a CI test must fail on any hosted macos-* runs-on. ... Route the #4719 and #4720 jobs through the owned Mac pool with a Blacksmith fallback. Run Linux-capable control and verdict work on Linux. Update the stale nightly-train guard and test that forbids Blacksmith." Leo: "THE GITHUB HOSTED MACS / ARE THE SLOWEST THINGS / THEY PREVIOUSLY WREN'T EVEN UNDER CONSIDERATINO" and "IT WAS ALWAYS FUCKING LAST RESORT" (Blacksmith macOS).

What changed

  • nightly-next-canary.yml (main's copy is the one the 15-minute schedule runs). Before, the canary Mac job ran only on blacksmith-6vcpu-macos-26. Now it takes the owned side label (vars.CI_SIDE_LANE_RUNNER) on attempt 1 while owned pools are on, and Blacksmith macOS 26 from attempt 2 on. pick and verdict already run on Linux.
  • nightly-next-train.yml: a new Linux pick-train-runner job runs scripts/ci/nightly_build_runner.py, the picker the nightly-next app build already uses. It picks an owned Mac (vars.NIGHTLY_NEXT_TRAIN_RUNNER, then vars.CI_SIDE_LANE_RUNNER) only while one of its runners is idle, and otherwise the Blacksmith macOS 26 label whose queue is moving. The compile reads the owned pick only on the pick's own attempt, so a re-run takes Blacksmith. Build mode runs from the nightly-train/* ref the drive pushes from feat-cmux-next, so the copy here is never executed. It changes only so main's owned-pool guard sees the workflow as owned on both branches (the feat-cmux-next PR follows this one).
  • owned_pool_rescue.py: the canary joins SIDE_WORKFLOW_PATHS, so a refused canary job is re-run onto Blacksmith. The train goes into UNRESCUED_WORKFLOW_PATHS, with the reason that its drive re-runs a refused compile itself (feat-cmux-next PR).
  • Stale guards
    • tests/test_ci_self_hosted_guard.sh no longer strips macos-15/macos-26 from *nightly*.yml and *notarization*.yml, and the background-lane check no longer skips those files. That exemption came from the retired 2026-10-19 "nightlies never run on Blacksmith" rule.
    • tests/test_ci_nightly_off_blacksmith.py drops two tests: one refused any non-literal Blacksmith mention, the other refused every shared runner variable. In their place it refuses a single-host label (cmux22, glaeda-runner-*), as feat-cmux-next's copy does. Its benchmark-mini rule matches feat-cmux-next's.
    • test_ci_no_github_hosted_macos.py's train check now asserts that vars.NIGHTLY_NEXT_TRAIN_RUNNER reaches only the picker's OWNED_LABELS, which offers only owned glaeda-* labels.

Why the hosted-macOS guard missed these

  • tests/test_ci_no_github_hosted_macos.py (landed in #26085, 2026-10-27 02:54 PDT) catches all three forms named in the HQ issues. Run on the pre-fix files, it reports:
    • the seed's {"pool": "hosted", "runner": "macos-26"} JSON;
    • the canary's bare runs-on: macos-26;
    • the train's ${{ vars.NIGHTLY_NEXT_TRAIN_RUNNER || 'macos-26' }}.
  • Those three lines were already gone from both branches (#26085 and #25887) before the research snapshot the issues cite.
  • What let them in before was the older shell guard:
    • check_no_github_hosted_runners and check_no_self_hosted_fleet_runners stripped macos-15|26 from every nightly file.
    • check_background_macos_lane skipped nightly files outright.
    • The runner-selection scan reads only runs-on:, os: and list lines, so a hosted label inside an env JSON string (the seed's SEED_POOLS, read through matrix.runner) was never seen.
  • The Python guard reads every non-comment line and does not exempt nightly files. This PR removes the shell guard's nightly exemptions, so neither guard has a nightly loophole now.

Before / after

  • Canary Mac job: before, blacksmith-6vcpu-macos-26 on every run. After, owned side label on attempt 1, Blacksmith on attempt 2+.
  • Train compile: before, vars.NIGHTLY_NEXT_TRAIN_RUNNER (unset), else blacksmith-6vcpu-macos-26. After, an idle owned Mac, else the moving Blacksmith label.
  • Repository variables: none changed.

Tests

These ran on a fast-checks box (run 40636515009, exit 0):

  • test_ci_no_github_hosted_macos.py, test_ci_nightly_off_blacksmith.py, test_ci_owned_pool_rescue.py, test_ci_owned_pool_sources.py (as main, against feat-cmux-next's tip)
  • test_ci_queue_janitor.py, test_ci_workflow_run_sources.py, test_runner_label_policy.py
  • the nightly-next canary and train tests, test_nightly_promotion_admission.py
  • test_ci_self_hosted_guard.sh

Risk: low.

  • The canary is advisory and holds no secrets. A refused or stuck owned job moves to Blacksmith by the side-lane rescue and the queue janitor.
  • The train is dry-run only (NIGHTLY_NEXT_TRAIN unset), and its compile runs from feat-cmux-next.
  • Open PRs that touch the same files: #26040 changes the canary's feed-origin line, not the runs-on line, so they don't conflict. #26614 changes nightly_build_runner.py, which this PR only calls, so it composes with this change.

Agent session: agent session

🤖 Generated with Claude Code

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by cubic

Fixes the main half of HQ issue 4720: GitHub-hosted macOS is never used, even as a fallback, and a CI test fails on any hosted macos-* runs-on.

  • The canary's Mac job takes the owned side label on attempt 1 and Blacksmith macOS 26 from attempt 2 on; fork PRs go to Blacksmith immediately.
  • The train compile gets a Linux picker that returns an idle owned Mac, else the moving Blacksmith label.
  • The owned-pool rescue treats the canary as a side lane (re-run onto Blacksmith) and the train as rescued by its own drive.
  • The self-hosted guard no longer exempts nightly and notarization files from the hosted-macOS and background-lane checks, and the stale Blacksmith-variable ban becomes a single-host-label ban.
  • The train workflow's copy on main is never executed (the drive builds from the nightly-train/* ref it pushes); it changes so the owned-pool guard sees both branches as owned.

Summary by CodeRabbit

  • Chores
    • Nightly macOS builds now select from available runner pools, with fallback routing when an owned runner is unavailable or a retry is needed.
    • Nightly workflows avoid routing builds to specific single-host runners and exclude GitHub-hosted Macs. These updates affect build infrastructure and do not change app functionality.