ci(queue janitor): page in-flight runs to their total_count, not to the first short page

Merged
TE
11d ago
+74-5

Summary

The queue janitor lists in-flight runs one status at a time (status=queued, in_progress, pending) and stopped at the first page with fewer than 100 runs. GitHub's status-filtered run listing can return a short page before the last one. At 06:10Z on 2026-10-26, status=queued returned 99 runs on page 1 and 85 on page 2, against a total_count of 182. So each sweep dropped every queued run older than page 1:

  • The pool snapshot undercounted the oldest waits. The 05:57Z macos-pool-load put blacksmith-12vcpu-macos-26's oldest wait at 24 minutes. Run 40259152940's fast tier had been queued there for 205 minutes by then. next_macos_cascade.py and pr_runner_pool.py route by those numbers, and the cascade's stall check reads that oldest wait.
  • Old runs were never acted on. Category b never cancelled run 40259152940, even though #25328 (merged at 6f07da41) made the janitor list cmux-next runs and PR #24021 had landed at 04:03Z. The 05:57Z sweep cancelled eight newer cmux-next stale runs and never saw this one. Its job finally got a Blacksmith runner at 06:08Z.

The fix: paging continues until a page is empty or the runs read reach that status's total_count. When there is no total_count, it falls back to the short-page rule. MAX_RUN_PAGES (3) still caps it, so a sweep costs at most one more request per status than before. owned_pool_rescue.GitHub.failed_runs() (status=failure) gets the same rule.

Risk: low. More runs are visible to the janitor, so each category may act on older runs it used to miss. Those runs were always in scope; the janitor just never saw them.

Testing

pytest -q tests/test_ci_queue_janitor.py tests/test_ci_owned_pool_rescue.py -k "InFlightRunListing or short_page" on cmux15 (Python 3.13):

  • base: 2 failed (test_a_short_first_page_does_not_hide_the_older_queued_runs, test_failed_runs_reads_past_a_short_page_while_the_total_says_more).
  • head: all pass. Across the janitor, rescue, starved-label and picker suites: 659 passed, 121 subtests passed.

Observed / Not run

  • Observed: the page sizes and total_count above, from the live API. The 05:57Z snapshot. That sweep's log (eight stale-pr cancels, no mention of 40259152940). Both test runs.
  • Not run: a live sweep with this code.

Changelog

none

Agent session: cosmic-gray

šŸ¤– Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by CodeRabbit

  • Bug Fixes
    • CI run listings now continue through short pages when more results may be available. Duplicate runs are listed only once, and listings stop when a page is empty or the page limit is reached.
  • Tests
    • Added coverage for pagination across short and empty pages, duplicate runs, page limits, and listings without a reported result count.