ci: keep the hosted light rescue off the last half of the GITHUB_TOKEN pool
Why
Leo asked: "why is there a repowide github api? how does THAT get counted?"
GitHub gives each repository one GITHUB_TOKEN pool: every workflow and job's github.token draws from the same installation limit. manaflow-ai is on the Free plan, so that pool holds 1,000 requests an hour (Enterprise Cloud would get 15,000). The pool has been running dry, and when it does the NEXT nightly tick (runs 41791114042 and 41791169626) and the notarize preflight (41790985234: gh: API rate limit exceeded for installation at 07:15Z) stall with it.
The biggest spender is the owned-pool sweeper's hosted-light pass (hosted_light_rescue.py), which runs from main every minute on HOSTED_LIGHT_TOKEN: github.token. Each pass:
- reads 12 workflows (the
hosted_light_jobsof feat-cmux-next's runners.json plus their callers) × 2 statuses × 2 listing pages, which is 48 listings. The listing URLs move with the time window each minute, so ETags never answer them. - reads the jobs of 40 of its 140–266 candidate runs.
- re-reads a recovery journal that always fails the same way ("2500 rescue runs in the window, more than 10 pages"), up to 10 more reads.
Sweeper run 41788968966 shows the result: from 07:06Z every listing and job read came back HTTP 403, 47 failures a minute.
Change (main only; the sweeper runs main's copy)
- Pool reserve. The client records
X-RateLimit-RemainingandX-RateLimit-Resetfrom every response, refusals and 304s included. Once the pool is down toPOOL_RESERVE(500), the pass makes no new listing or job read until the reset. The other half stays for the nightly and the other workflows. Recovery of a cancellation the pass already owns still finishes, and so do writes. - Overflow page only when needed. The pass now reads the rotating overflow listing page only when the newest page is full (100 runs). Today it reads that page every time, even when it comes back empty.
- Journal retry. An unreadable recovery journal is read again after 15 minutes instead of every minute.
The workflow, its token, and its permissions and secrets are unchanged. CI_HOSTED_LIGHT_RESCUE_DISABLED=1 still turns the pass off, and reverting this PR restores the old behaviour.
Performance
| Metric | Before | After | Delta | Harness or run |
|---|---|---|---|---|
| GITHUB_TOKEN requests per pass, listings (12 workflows) | 48 | 24 (48 only for a status with over 100 runs) | −50% | Request count from the code; candidate counts from sweeper run 41788968966 |
| GITHUB_TOKEN requests per hour, failing journal re-read | up to 600 | up to 40 | −93% | Same run: the error repeats every pass |
| GITHUB_TOKEN pool the pass can spend in an hour | all 1,000 (spent within minutes of each reset) | at most the 500 above the reserve | at least 500 kept for other workflows | Same run, 403s from 07:06Z; Quota tests |
| Hosted-light pass, GITHUB_TOKEN requests per hour (estimate) | about 3,000–5,900 attempted | at most about 500 used | −80% or more | Request count from the code, at 1–2 passes a minute |
This job compiles nothing, so build caches don't apply.
Tests
New Quota cases in tests/test_ci_hosted_light_rescue.py cover four behaviours:
- no overflow read when the newest page is not full
- no listing or job read at the reserve
- job reads stop partway through a pass when the pool reaches the reserve
- the client reads the pool from 200 and 403 responses, and the pool counts as spare again after the reset
A fifth case checks that the journal is retried only after 15 minutes. The suite passes locally (53 tests). On the fleet, run 41795804793 ran tests.test_ci_hosted_light_rescue, tests.test_ci_owned_pool_rescue and tests.test_ci_runner_pools at 0913814bfcc472a2b2f15b22ffa5d68097f00a27: 567 tests, OK.
Prior art: #34508 moved these reads off the routing App onto github.token, and #34807, #34820, #34571 and #35003 changed the scan and recovery. This PR keeps their token choice and rotation and only bounds the spend.
🤖 Generated with Claude Code
Agent session: agent session
Summary by CodeRabbit
- Improvements
- Hosted recovery scans now avoid requesting an extra listing page when the latest page is not full.
- When request capacity is low, scans pause new listings and job reads while allowing recovery already in progress to continue. Scans can resume after the request limit resets.
- Failed recovery-journal reads are retried after a delay rather than on every scan pass.