PR #32023 - manaflow-ai/cmux

What

A declared hosted light job (.github/runners.json hosted_light_jobs: routers, resolvers and report steps, cmux#31785 and cmux#31878) takes GitHub-hosted ubuntu-24.04 on attempt 1. Its fallback (RUNNER_POOLS.linux) applies only on a re-run, and today nothing starts that re-run:

ci-hosted-queue-rescue.yml covers only CI, CLA and CI fast guards, and only after GitHub cancels the whole run (about 15 minutes);
the queue janitor's lost-assignment pass waits 30 minutes, needs proof that a newer job on the same labels was taken, and does not list feat-cmux-next's Linux-only workflows.

The rulebook asks for a stall-aware failover after about 3 minutes.

scripts/ci/hosted_light_rescue.py is the feat-side entry point. Main's overflow watch (ci-cloud-overflow-probe.yml) will run it at a pinned commit in a follow-up PR on main, the same way it runs linux_pool_spill.py (#31884 then #31912). Each pass:

lists the in-flight pull request runs of every workflow that declares a hosted light job, plus the workflows that call the reusable ones (pr-red-checks-comment.yml and ci-guards.yml run inside cmux-next.yml, cmux-next-guards.yml and ci.yml). It lists runs created between 3 and 90 minutes ago, one page per workflow and status (14 listings);
reads the jobs of the oldest 40 runs that qualify: attempt 1, and at the PR's current head;
picks a run when its only unfinished jobs are stalled. A stalled job is queued with no runner id or name, every label is ubuntu-*, and it has waited 3 minutes or more. It then cancels the run, waits for the cancel to settle (90 s budget per pass), and calls rerun-failed-jobs. The re-run is attempt 2, so the jobs take their fallback route. Jobs that already passed keep their result.

Bounds: pull request runs only, so never push, dispatch, schedule, nightly, DEV, promote or release runs. nightly.yml is excluded outright. Attempt 1 only, so a job fails over at most once. At most 10 re-runs per pass. A run with any other unfinished job is reported and left alone. A cancel that does not settle in time is reported and is never re-run blind later, because a later pass cannot tell our cancel apart from a person's or a concurrency group's. Dry run is supported (DRY_RUN).

Measured first: the hosted queue is not real right now

The dashboard's pools showed ubuntu-24.04 with 27 queued, the oldest about 2,900 s (08:25Z on 2026-11-02). I read each of those 27 jobs from the REST jobs endpoint:

26 had completed; the 27th was in progress;
runner wait (started_at - created_at): 1 to 94 s, median about 10 s;
one job had runner 0 and was cancelled 7 s in, by its concurrency group.

The dashboard keeps finished jobs in queued, so this is staleness (the HQ ci-dash issue is already filed). The point of this PR is the fallback guarantee if the hosted cap is ever hit, not a live backlog.

Performance

metric	before	after	delta	harness or run link
hosted light job runner wait, 27 dashboard-"queued" jobs at 07:36–08:25Z	1–94 s (median ~10 s)	unchanged when hosted serves	0	REST `actions/jobs/{id}` for each job the dashboard listed
worst-case stall of a hosted light job GitHub never assigns	~15 min, then a cancelled required check (CI only; none for the other 6 workflows)	3 min plus the overflow watch interval (10–18 min observed), then attempt 2 on the fallback route	earlier, and covers all 7 workflows	`ci-cloud-overflow-probe.yml` run history
API reads per watch pass	0	≤ 14 listings + ≤ 40 job reads + settle reads	+≤ ~60	`test_re_runs_and_job_reads_are_bounded`
Blacksmith minutes/day	unchanged	+ only the re-run of a stalled light job (2–50 s each)	~0	—

Validation

python3 -m unittest tests.test_ci_hosted_light_rescue tests.test_ci_linux_pool_spill: 35 OK (4 skipped, the spill's main-policy cases).
bash tests/test_ci_self_hosted_guard.sh: pass.
python3 scripts/ci/validate_test_execution_registry.py: valid. The new test is registered manual with a reason (no new CI gate).

Observed / Not run

I ran the commands above and read the 27 live jobs. Not run: a live pass. That needs main's wiring PR, which pins this PR's merge commit.

Changelog

none

Agent session: agent session

🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

Adds scripts/ci/hosted_light_rescue.py so a declared hosted light job that GitHub-hosted Ubuntu never assigns gets failed over to its re-run route after about 3 minutes, instead of waiting up to 30 minutes for the queue janitor or 15 for the whole-run cancel.

Scans in-flight pull request runs of workflows that declare hosted light jobs (plus callers of the reusable ones) and cancels a run whose only unfinished jobs are stalled on ubuntu-* labels with no runner for 3+ minutes, then re-runs the failed jobs on attempt 2, which takes the declared fallback route.
Bounded to PR runs at attempt 1 at the PR's current head; reads at most 40 runs and re-runs at most 10 per pass; skips a run with any other unfinished job or whose jobs span more than one page; and no cancel once the pass's 90 s settle budget is spent.
Before cancelling, re-reads the PR head and the run's jobs so a PR that moved on or a job a runner took meanwhile is left alone; a cancel that never settles is left unre-run.
Main's ci-cloud-overflow-probe.yml will run the script at a pinned commit in a separate wiring PR; the new tests are registered manual and listed as unwired, so they add no CI gate.