ci(guards): each runner guard check runs and is compared on its own; chief-evals on Blacksmith

Merged
TE
10d ago
+343-54

Leo (2026-10-27): "When I ask you guys to do things, I'm not just saying do this individual instance. I'm saying go do stuff."

Problem

tests/test_ci_self_hosted_guard.sh runs ~30 runner checks as one CI step under set -e. Since #25610 put chief-evals.yml on hosted ubuntu-24.04 (2026-10-26), that step has been red on the base. The base-red machinery (guards_new_reds.py) compares per step, so the whole step became a base warning. Also, under set -e every check after the first failure stopped running (check_no_self_hosted_fleet_runners, check_background_macos_lane and ~25 others). Any new hosted or self-hosted runner line could land as a warning.

Change

  • run_check wraps every check: each runs in its own subshell, the script keeps going, fails once at the end, lists the failed checks, and appends per-check outcomes to $CMUX_GUARD_CHECK_RESULTS.
  • run_ci_guards.py sets that file per step and adds "<step> :: <check>" entries to --results / --json.
  • guards_new_reds.py compares those per check. A base from before this change has only the step result, which stands in for each check.
  • Each failed check, and each of its FAIL: findings, also prints a plain error: line. gh-merge-green's base_red_excuse.py compares a red step's error lines with the base's, so that path now compares per check too.
  • tests/test_guards_new_reds.py (runs in cmux-next.yml) gets new tests: a fixture with two checks, one red on the base, where the new red fails and the base red stays a warning; and a base from before the split, which still compares. The new tests error out against the old run_ci_guards.py.
  • New tests/test_ci_guard_check_isolation.py (preflight group; the same file lands on main in #26398):
    • every check still runs after an earlier one fails;
    • every check_* function is called through run_check;
    • the per-check error lines stop a base red from excusing a different check.
      Against the old guard script, 6 of 6 tests fail.
  • chief-evals.yml: runs-on moves to blacksmith-4vcpu-ubuntu-2404 (hosted only in forks, per the repo convention), and the design note is updated.

Checks the masking had hidden (now fixed here)

Before: 1 check ran red and the rest after it never ran. After, on the old tree, 3 of 30 were red:

  • check_no_github_hosted_runners: chief-evals.yml:35. Fixed above.
  • check_gui_smoke_unsupported_launch_handling: still required launch smokes in release.yml, which #24621 removed on purpose ("we want no smoke tests"). The check drops the release.yml requirement and keeps the helper and nightly checks.
  • check_pr_macos_workflows_cancel_superseded_runs: cmux-next-ios-kit.yml used cancel-in-progress: ${{ github.ref != 'refs/heads/feat-cmux-next' }}, which the guard cannot read as cancelling pull request runs. Now ${{ github.event_name == 'pull_request' }}. Its push trigger is feat-cmux-next only, so behaviour is the same apart from dispatches.

After: 30/30 checks pass, and tests/test_guards_new_reds.py 10/10 and tests/test_ci_guard_check_isolation.py 6/6 pass.

Risk: low. This is CI-only. A check that exits now ends only its own subshell.

Agent session: agent session

šŸ¤– Generated with Claude Code