#35316 ci: the overflow watch reads as cmux-bot, the workflow token only for writes and refused reads

Leo's coordinator asked to cut the repository token's calls per hour so the NEXT nightly stops failing on "API rate limit exceeded for installation" (run 41790985234). Every workflow's GITHUB_TOKEN shares one 1,000/h pool.

The overflow watch (ci-cloud-overflow-probe.yml, every 10 minutes) reads with github.token:

• It reads this run's probe jobs every 15 s until each probe starts or waits CI_CLOUD_PROBE_MINUTES, with up to 5 minutes of grace after that.
• When a probe stalls, it lists queued and in-progress runs and reads up to 60 of them for their jobs.

These reads now use a cmux-bot installation token, which has its own pool:

• New step. The watch job mints the cmux-bot token with permission-actions: read only, from the CMUX_BOT_APP_ID/CMUX_BOT_APP_KEY that other trusted jobs already use. The step is continue-on-error, and the watch is a schedule/dispatch job, never pull_request.
• Token wiring. GH_TOKEN is the bot token || github.token, and the new GH_FALLBACK_TOKEN is github.token.
• cloud_overflow_switch.GitHub. With a fallback, GETs go out on the primary token. A GET refused with 401, 403 or 429 is asked once more on the workflow token; a refused GET changed nothing, so the retry is safe. Writes (force-cancel and re-run) always use the workflow token, so who cancels and re-runs is unchanged. With no fallback, it behaves as before.
• Unchanged. Variable writes stay on the routing App's switch token. The runner-pools step and the stuck-probe self-cancel stay on github.token; they make about 2-3 calls a run.

No new secret or permission.

Performance

metric	before	after	delta	harness or run link
watch reads on the repository GITHUB_TOKEN pool, Blacksmith serving (probes start within ~30 s)	~3 a run, ~20-30/h	0	about -25/h	static read of wait_for_probe (15 s poll), 6 runs/h
the same while a probe stalls (5 min wait + grace, then rescue)	~20-40 polls + 2 listings + up to 60 job reads a run, up to ~500/h	0	up to about -500/h	static read of wait_for_probe and rescue_stuck_runs (MAX_RUN_LISTINGS 60)
force-cancel and re-run writes	on the workflow token	unchanged	0	ReadTokenTests

During a Blacksmith stall the watch was one of the heaviest scheduled readers on the pool, at the very moment the rescues and re-runs need it most.

Tests

• python3 tests/test_ci_cloud_overflow_switch.py (67 passed). New ReadTokenTests cover:
  • reads use the bot token, and writes use the workflow token;
  • a 401, 403 or 429 read is retried once, and a 404 is not;
  • with no fallback, everything uses the one token, as before;
  • the workflow wiring: a read-only mint that comes before the switch step, the fallback wiring, and no routing App.
• tests/test_ci_runner_pools.py (45), tests/test_ci_repo_variable_defaults.py, tests/test_ci_fork_runner_routing.py and tests/test_ci_self_hosted_guard.sh pass.
• actionlint is clean.

Prior art: #35264 (the same split for the hosted light rescue, merged), #34508 and #34200 (this watch's token and runner).

Changelog

none

🤖 Generated with Claude Code

Agent session: agent session

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Moves the overflow watch's read calls onto a read-only cmux-bot installation token, so they stop consuming the repository GITHUB_TOKEN's 1,000/hour pool shared by every workflow, which caused the nightly's "API rate limit exceeded" failure.

• Force-cancel and re-run stay on the workflow token, and a read refused with 401, 403 or 429 is retried once on it.
• Variable writes still use the routing App's switch token; no new secret or permission is added, and without a fallback everything behaves as before.

Summary by CodeRabbit

• Bug Fixes
  • Cloud overflow checks can now retry read requests with the workflow’s existing credentials when the primary credentials are rejected or rate-limited.
  • Write operations continue using the workflow’s existing credentials.
  • Other read errors are not retried.