ci(fast-checks): take the overflow at once when the routing quota is spent
Leo-ask: ask-6ef2624788
What happened
Fast-checks runs 41614121649 (#33537), 41614232965 (#33562) and 41614590891 (#33564) each sat in admit ā "Select a live fleet runner or bounded overflow" for 180-181 s, then took blacksmith-8vcpu-ubuntu-2404. The step did not hang. Every census read in all three logs is HTTP 403 X-RateLimit-Remaining=0 X-RateLimit-Resource=core for the routing App's installation. The picker asked again every 15 s (13 refusals per run) until its 180 s deadline ran out. The warning it printed blamed a missing "Self-hosted runners: read" permission, which was wrong.
Change
pr_runner_pool.rate_limited(error)recognizes GitHub's quota refusal: a 429, or a 403 withX-RateLimit-Remaining: 0orRetry-After. It also checks any error the refusal was raised from. On that refusal,GitHub.runners()raisesRateLimited, aRuntimeErrorsubclass, so existing callers keep their fallback, instead of the permission message.fast_checks_runner: a spent quota takes the measured overflow at once. Polling again would only be refused again and would burn more of the shared routing quota. A plain permission 403 still polls up to the deadline.- Whole-admission bound: feat-cmux-next already passes one deadline through
read_liveand caps every page read with it (#30762, #30822). New test: every census page drips bytes, andchoose()still returns within its deadline.
Red/green
Run with python3 tests/test_ci_fast_checks_runner.py (19 tests, OK); tests/test_ci_pr_runner_pool.py also passes (262 tests, OK).
- Base feat-cmux-next: the 4 new spent-quota tests fail; the drip test passes, because the deadline already exists here.
- Pinned
ecc5d6074e78helper, which main's fast-checks.yml still runs: the drip test fails (3.08 s not less than 0.8), and the spent-quota test fails with 12 polls of 15 s.
A follow-up PR on main bumps that pin to this merge. It also adds github_spend.py to the sparse checkout, passes CI_RUNNER_STATE, and adds a job-level timeout-minutes to admit.
Risk: CI picker only. A wrong rate-limit match sends a run to Blacksmith overflow early, which is the same place it lands after 180 s today.
š¤ Generated with Claude Code
Agent session: agent session
![Autofix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-dark-3x.png)
Open in [code]smith Ā· Need help on this PR? Tag @codesmith-bot with what you need.
Summary by cubic
Fast-checks admission jobs polled for the full 180 s before overflowing when the routing token's GitHub quota was spent, because every census read was refused. This change recognizes a spent quota (a 429, or a 403 with X-RateLimit-Remaining: 0 or Retry-After) and takes the measured overflow immediately instead of polling, which only got refused again and burned more of the shared routing quota. A plain permission 403 still polls until the deadline.
GitHub.runners()now raisesRateLimited(aRuntimeErrorsubclass) on quota refusals so existing callers keep their fallback; the check walks the error's cause chain.- Adds tests covering the new overflow paths and a whole-admission deadline that holds even when every census page drips bytes.