ci(queue janitor): see cmux-next runs; move a cmux-next leg stuck on Blacksmith macOS

Merged
TE
11d ago
+434-55

Summary

cmux-next run 40259152940 (PR #24021) placed its cmux-next fast tier job on blacksmith-12vcpu-macos-26 at 02:32Z on 2026-10-26. The job sat queued for more than 90 minutes with no runner. Two gaps in the queue janitor let that happen. This PR fixes both.

1. The cascade read the Blacksmith queue wrong. next_macos_cascade.py on feat-cmux-next counts each Blacksmith label's queue and its oldest wait (its stall check) from the janitor's macos-pool-load snapshot. The janitor runs from main. On main, cmux-next.yml is an inert workflow_dispatch registration with no macOS text, so linux_only_workflow_paths() reads it as Linux-only, and needs_jobs() never listed a cmux-next run's jobs. Only the starved-label alert read them. The snapshot therefore missed every cmux-next job waiting on Blacksmith macOS:

  • The placement log at 02:32:46Z says: "blacksmith-12vcpu-macos-26 1 in flight + 2 queued of 5 ... takes fast-tier". Job listings show about ten cmux-next jobs already queued on that label then, with no runner (runs 40257975146, 40258172516, 40258191884, 40255513637, 40255836148, 40257927928, 40255978446, 40258410547, 40259009871, ...).
  • The 03:51Z snapshot gives the 12 vCPU label's oldest wait as 37 minutes. Our job had been waiting 79 minutes by then.
  • With those jobs counted, the label had no free slot and was past the cascade's 25-minute stall wait. The cascade would have sent the fast tier to hosted macos-26, which had 0 of its 5 slots in use.

Fix: follows_another_copy(). A run into or on a branch other than main follows that branch's copy, so it is listed even when main's copy is Linux-only. So is a same-repository pull request run that no longer names a pull request: GitHub drops the PR once it closes, as happened to run 40259152940 when #24021 landed. Category b then cancels it as stale. The starved-label pass already listed most of these runs on every sweep, so the API cost stays about the same. starved_scope_runs() is removed.

2. Nothing moved a job queued on Blacksmith. owned_pool_rescue.py moves only owned (glaeda-*) legs, and category f only covered owned legs too. next_macos_cascade.py itself says "nothing rescues a job queued on Blacksmith". Category f now also takes a cascade job (fast tier, Swift canary, Release compile, scheme compile) of a same-repository cmux-next pull request run, queued on a blacksmith-*-macos-* label for more than 25 minutes (BLACKSMITH_LEG_SECONDS, the cascade's own STALL_MINUTES), while owned pools are on. It cancels the run, and a later sweep re-runs the failed jobs, as for an owned leg. The re-run is made only with an App's token. In cmux-next.yml, a re-run that github-actions[bot] did not trigger goes to needs.macos-placement.outputs.runner, the owned side label the rescue watches. That bot's attempt 2 would take attempt 1's Blacksmith placement again. How that is enforced:

  • No App token this sweep: the leg is skipped.
  • A pending entry is marked app_actor and waits for an App token.
  • rerun_failed(app_only=True) never falls back to GITHUB_TOKEN.
  • Nothing is moved past the rescue's ladder (REFUSAL_RESCUE_ATTEMPT), so the janitor adds at most one attempt.
  • The existing rules for category f still apply: a live head, no unfinished sibling, the run not already failed, no-janitor.
  • Any cmux-next run holding an unassigned Blacksmith leg is re-run only by an App, whichever category cancels it (category e included).
  • A Blacksmith leg in a run category e holds but does not cancel (a bot's attempt) is judged by category f.
  • GUI and session jobs are left out: with no GUI or session runner online, their re-run still takes blacksmith-6vcpu-macos-26. Fork runs are left out because their runs-on takes Blacksmith first.
  • The orphan pass now also leaves feat-cmux-next pushes alone, since they are listed now and gate the nightly-next build.

Is this systemic? Yes. I listed the 287 cmux-next pull request and push runs that ran over 25 minutes since about 10:00Z on 10-25. In them, 58 of 110 jobs sent to blacksmith-12vcpu-macos-26 and 52 of 146 sent to blacksmith-6vcpu-macos-26 never got a runner. Of the ones that did start, the 12 vCPU label's p90 wait was 37 minutes (max 54) and the 6 vCPU label's was 55 (max 77). From 02:14Z on, fast-tier jobs on the 12 vCPU label sat 30 to 87 minutes with no runner until something cancelled them. Release compiles were still queued after 110 minutes.

Risk: medium. cmux-next pull request runs into feat-cmux-next now go through every janitor category. In practice that means:

  • b: cancels closed or superseded runs.
  • e: cancels lost assignments.
  • f: cancels an owned leg unclaimed past its bound, and now a Blacksmith leg as above.

feat-cmux-next pushes stay protected (PROTECTED_PUSH_BRANCHES). The snapshot's counts for Blacksmith and owned labels now include cmux-next's jobs. That makes ci.yml's picker and cmux-next's cascade see the real load, so either may route differently.

Testing

../venv/bin/python -m pytest -q tests/test_ci_queue_janitor.py on cmux15 (Python 3.13, PyYAML):

  • base (origin/main script with the new tests): 15 failed (the new BranchCopyListingTests and BlacksmithLegTests).
  • head: 204 passed, 108 subtests passed.

Review: Codex (gpt xhigh, read-only) over four rounds. Rounds 1 to 3 found the category-e App bypass, the fork, GUI/session and dispatch scope, the orphan pass on feat-cmux-next pushes, and the e/f overlap for a bot's attempt. All are fixed. Round 4 gave SHIP with no findings at 2be77bc6.

pytest over test_ci_health_report.py, test_ci_owned_pool_rescue.py, test_ci_owned_warm_state.py, test_ci_pr_runner_pool.py, test_ci_queue_janitor.py and test_ci_starved_labels.py: 743 passed (656 across the four janitor/rescue/starved/picker suites at the final head). 4 errors, because the scratch copy has no docs/ for the health report's documentation tests.

Observed / Not run

  • Observed: the placement log and job listings for run 40259152940, 96 janitor snapshots from the last 24 hours, and job listings for 287 cmux-next runs (numbers above). Both test runs above.
  • Not run: a live janitor sweep with this code. The behaviour for runs-on and the triggering actor is read from feat-cmux-next's cmux-next.yml, not exercised.

Changelog

none

Agent session: cosmic-gray

🤖 Generated with Claude Code