ci(fast-checks): take the overflow at once when the routing quota is spent

Merged
TE
4d ago
+144-4

Leo-ask: ask-6ef2624788

What happened

Fast-checks runs 41614121649 (#33537), 41614232965 (#33562) and 41614590891 (#33564) each sat in admit → "Select a live fleet runner or bounded overflow" for 180-181 s, then took blacksmith-8vcpu-ubuntu-2404. The step did not hang. Every census read in all three logs is HTTP 403 X-RateLimit-Remaining=0 X-RateLimit-Resource=core for the routing App's installation. The picker asked again every 15 s (13 refusals per run) until its 180 s deadline ran out. The warning it printed blamed a missing "Self-hosted runners: read" permission, which was wrong.

Change

  • pr_runner_pool.rate_limited(error) recognizes GitHub's quota refusal: a 429, or a 403 with X-RateLimit-Remaining: 0 or Retry-After. It also checks any error the refusal was raised from. On that refusal, GitHub.runners() raises RateLimited, a RuntimeError subclass, so existing callers keep their fallback, instead of the permission message.
  • fast_checks_runner: a spent quota takes the measured overflow at once. Polling again would only be refused again and would burn more of the shared routing quota. A plain permission 403 still polls up to the deadline.
  • Whole-admission bound: feat-cmux-next already passes one deadline through read_live and caps every page read with it (#30762, #30822). New test: every census page drips bytes, and choose() still returns within its deadline.

Red/green

Run with python3 tests/test_ci_fast_checks_runner.py (19 tests, OK); tests/test_ci_pr_runner_pool.py also passes (262 tests, OK).

  • Base feat-cmux-next: the 4 new spent-quota tests fail; the drip test passes, because the deadline already exists here.
  • Pinned ecc5d6074e78 helper, which main's fast-checks.yml still runs: the drip test fails (3.08 s not less than 0.8), and the spent-quota test fails with 12 polls of 15 s.

A follow-up PR on main bumps that pin to this merge. It also adds github_spend.py to the sparse checkout, passes CI_RUNNER_STATE, and adds a job-level timeout-minutes to admit.

Risk: CI picker only. A wrong rate-limit match sends a run to Blacksmith overflow early, which is the same place it lands after 180 s today.

šŸ¤– Generated with Claude Code

Agent session: agent session


Autofix with [code]smith
Open in [code]smith Ā· Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fast-checks admission jobs polled for the full 180 s before overflowing when the routing token's GitHub quota was spent, because every census read was refused. This change recognizes a spent quota (a 429, or a 403 with X-RateLimit-Remaining: 0 or Retry-After) and takes the measured overflow immediately instead of polling, which only got refused again and burned more of the shared routing quota. A plain permission 403 still polls until the deadline.

  • GitHub.runners() now raises RateLimited (a RuntimeError subclass) on quota refusals so existing callers keep their fallback; the check walks the error's cause chain.
  • Adds tests covering the new overflow paths and a whole-admission deadline that holds even when every census page drips bytes.