PR #28402 - manaflow-ai/cmux

Summary

The queue janitor (scripts/ci/queue_janitor.py, run from main) listed at most three pages of runs per status (MAX_RUN_PAGES = 3). On 2026-10-30 about 1,138 runs were queued, so a sweep saw 300 of them. Every run past the cap was missing from the macos-pool-load snapshot (pools, reserved_queued, label_started), which the pickers, the rescue's failover, the cascade and the fleet dashboard read. The cleanup categories never saw those runs either.

Listing. Each status is paged until an empty page. GitHub returns at most 1,000 results for a filtered listing, so a status that fills ten pages is listed again from the oldest run seen (created<= that time, newest first), one window at a time. The ceiling is 40 pages per status (4,000 runs, about 3.5 times today's peak). Job listings run six at a time, newest first, with at most 2,000 per sweep. Done one at a time, the full queue would not fit the workflow's 10-minute timeout.
Snapshot. The snapshot now writes truncated, listed and total, plus listing.statuses (per status: listed, total_count, pages, truncated) and listing.jobs_unlisted. A truncated sweep:
- decides nothing, the same way an unreadable run already makes it decide nothing;
- sets starved_incomplete, so no starved-label issue is opened or closed;
- emits a ::warning.
Readers. pr_runner_pool.snapshot_truncated() feeds snapshot_problem(), so every reader that uses it treats a truncated snapshot as unknown, never as "nothing queued": simple_pool_picker, the rescue's pool_snapshot() and the classic nightly runner. decide() keeps the default. e2e_runner_pool.measure_load() (E2E and iOS) returns None. runner_pools.label_evidence() records no snapshot evidence. Any value other than false counts as truncated. A snapshot without the key (an older janitor) reads as before.
Sweeper cancelled before it ran. Run 41071107039 (the cron sweeper, 14:21Z) was cancelled at 14:56:46Z. Three seconds earlier, the running sweeper's code handover (main's ... changed) dispatched run 41076139116 into the same owned-pool-sweeper concurrency group. That group never cancels the running sweeper, but holds only one pending run, so the newer pending dispatch evicted the older pending cron run. Run 41055530432 was evicted the same way at 14:08Z (by dispatch 41069145800), and so were most scheduled sweepers since 10-28. Now a handover first lists pending and queued sweeper runs (two requests, made only at a handover) and dispatches only when none is waiting. The waiting run checks out main when it starts, so it already runs the new code. If the listing fails, it dispatches as before.

Performance

Replayed with the janitor's own listing and read_run_jobs against the real in-flight listing captured at about 15:30Z on 2026-10-30 (882 queued, 234 in progress, 57 pending). "Before" is main's three-pages-per-status listing over the same capture. Sweep wall time "before" is the live sweep 41076464078 at 15:03Z (592 runs, 620 calls). "After" wall time is an estimate from that sweep's per-request time; the first sweep after merge measures it (stop condition below).

metric	before	after	delta	harness or run link
in-flight runs listed	582 of 1,173	1,163 of 1,173 (the rest moved between pages)	+581	replay of the captured listing
queued runs listed	292 of 882	873 of 882	+581	replay
run-listing requests	8	16	+8	replay
job-listing requests	550	1,093	+543	replay
API requests per sweep (listing + jobs)	558	1,109	+551 (about 1 per in-flight run, capped at 2,000 + 120)	replay
sweep wall time	297 s (620 calls, one at a time)	about 100 s estimated (1,109 calls, 6 at a time)	about -200 s	https://github.com/manaflow-ai/cmux/actions/runs/41076464078

Cost bound: at most 3 × 40 run-listing pages plus MAX_JOB_LISTINGS = 2,000 job listings per sweep. The sweep runs on the glaeda-route App token, at about 6 sweeps an hour.

Tests (red on main, green here)

The first commit adds the tests alone. Run on main's code they give FAILED (failures=9, errors=8) over 13 tests:

FullQueueListingTests: a 1,138-run queue lists every run; a listing past its ceiling says truncated with counts; runs past the job-listing cap truncate; parallel listings match serial ones; a rate limit still ends the sweep.
TruncatedSweepTests: a complete sweep says so and acts; a truncated one marks the snapshot and cancels nothing.
TruncatedSnapshotTests (pr_runner_pool): snapshot_problem, decide and the E2E/iOS load treat truncated as unknown.
TruncatedSnapshot (runner_pools): no evidence from a truncated snapshot.
Sweeper.test_a_handover_dispatches_no_sweeper_when_one_already_waits and test_waiting_sweeper_finds_....

The existing test_only_an_empty_page_or_the_page_cap_ends_a_status asserted the 300-run cap and now asserts four full pages.

Observed / Not run

Ran python3 -m unittest for test_ci_queue_janitor (260), test_ci_pr_runner_pool (255), test_ci_owned_pool_rescue (405), test_ci_runner_pools (28), test_ci_simple_pool_picker and test_ci_starved_labels: all pass. test_ci_repo_variable_defaults, test_ci_workflow_run_sources, test_nightly_delivery_concurrency and test_cmux_next_dev_tip_feed pass. scripts/verify-local.py passes. test_ci_change_areas fails the same 5 tests on main in this environment (they shell out to gh).
Not run: a live sweep on this head (the workflow runs from main); the measured after wall time comes from the first sweep after merge.

Changelog

none

Prior art: #25356 (paged to an empty page, kept the 3-page cap), #28312 and #28374 (snapshot gates, delivers()). Read #27114 (held; dead-label redispatch), #28249, #28145 and #28352: none changes the run listing's cap or the sweeper handover.

Agent session: agent session

🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

The queue janitor (scripts/ci/queue_janitor.py) previously listed at most three pages of runs per status, so with 1,138 runs queued on 2026-10-30 it saw 300, and the macos-pool-load snapshot read every other run's jobs as nothing queued. It now pages each status until an empty page (up to 40 pages, past GitHub's 1,000-result cap), lists jobs six at a time, and records truncated, listed, total, and per-status listing data in the snapshot. Job listings sent with a just-refused App token now retry on the workflow token.

A truncated sweep decides nothing and opens or closes no starved-label issue; pool readers (pr_runner_pool, the rescue's failover, E2E/iOS load, runner-pool evidence) treat a truncated snapshot as unknown, never as an empty queue.
The rescue sweeper's code handover now lists pending and queued sweeper runs and dispatches only when none waits, so the cron's pending sweeper is no longer cancelled before it starts.

Summary by CodeRabbit

Bug Fixes
- CI runner and queue management now reject incomplete or truncated run listings when evaluating loads, labels, and queue actions.
- Sweeper handovers skip dispatching a replacement when another eligible sweeper is already waiting.
- Queue listings continue across result pages, while job listings are capped and fetched concurrently. Rate-limit errors still stop the sweep.
Documentation
- Updated the queue sweep API-cost estimate with its calculation, observed run count, date, and shared-token budget context.