ci(queue janitor): retry a read on a 5xx; one unreadable run skips only that run

Merged
TE
25d ago
+295-10

Problem

At 08:08Z on 2026-10-26 the queue janitor (run 40290134950) failed on a single GET /actions/runs/40288637818/jobs that returned 502. GitHub.request() did not retry it, and any failed job listing ended the sweep before it wrote the macOS pool snapshot.

The snapshot went stale; the newest was from 07:55Z. The cmux-next cascade reads that snapshot, so it then sent no job to Blacksmith ("no pool snapshot under 15 min old to count Blacksmith's queue; no job leaves the minis' queue"). The feat-cmux-next push Release compile 121466224653 (run 40291067104, the #25360 merge) was left queued on the busy glaeda-side-std-xcode-26.6, and nightly-next can't promote without it.

Fix

  • Retry reads. GitHub.request() repeats a read (a GET, or a GraphQL query) after 2 s and then 6 s (RETRY_DELAYS) when it gets a 500, 502, 503 or 504, or a dropped connection. After that it fails as before. A write (cancel, re-run, dispatch) is never repeated, since the first attempt may have taken effect.
  • One unreadable run no longer ends the sweep. If a run's job listing still fails, read_run_jobs() leaves that run out and the snapshot is still written from every other run. The snapshot is marked starved_incomplete (so no starved-label issue is closed as recovered) and names the run in unread_runs. That sweep then decides nothing: no cancel, re-run or orphan cancel. A partial inventory would misjudge the other runs too (a newer run at the same head, the pool's wait p90). Pending re-runs carry forward unchanged.
  • Truncated responses are retried too. A dropped or truncated response, while opening it or reading it (http.client errors, resets, timeouts), is retried like a 5xx.
  • Rate limits still stop the sweep. A rate limit raises RateLimited and ends the sweep, as before.

Risk

  • A skipped run's demand is missing from that one sweep's snapshot, and that sweep cancels nothing. The next sweep, 10 minutes later, reads it again and acts as before.
  • The worst case adds 8 s per failing read.
  • This replaces test_a_failed_listing_of_a_branch_run_fails_the_sweep_like_any_other, which pinned the old stop-on-any-failure behaviour, with test_a_failed_listing_skips_its_run_and_marks_the_snapshot_partial.

Tests

tests/test_ci_queue_janitor.py::TransientFailureTests:

  • a read is made again after a 502 and after a dropped connection, with the configured delays
  • the retries are bounded
  • a POST write is not repeated, while a GraphQL query is
  • truncated and dropped responses, while opening and while reading, are made again
  • with one unreadable run the snapshot is still written and names it, the ghost run a full sweep cancels is not cancelled, and a pending re-run is carried forward (OrphanSweepTests.test_an_unreadable_run_writes_the_snapshot_and_cancels_nothing; turning the guard off fails it)
  • a rate limit still ends the sweep

On cmux15, test_ci_queue_janitor.py, test_ci_starved_labels.py and test_ci_owned_pool_rescue.py all pass: 418 tests.

Agent session: agent session

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • CI queue checks now retry temporary failures when reading run and job information, including server errors and connection or timeout issues. Read-only GraphQL requests are also retried; write requests are not.
    • When job details can’t be retrieved, the sweep records the affected runs in an incomplete snapshot and skips cancellation, rerun, and orphan-processing actions.
    • Rate-limit failures continue to stop the sweep rather than being treated as ordinary read errors.