ci(tick): the nightly-next tick waits through GitHub API rate limits

Merged
TE
11d ago
+436-1

The nightly-next tick now waits out a GitHub API rate limit instead of failing, and its first refusal logs which limiter refused it.

Why. Tick 40325957281 (12:53Z) failed with "gh: API rate limit exceeded for installation" and lost its promotion, even though the token's /rate_limit showed thousands of calls left. The tick runs from main, so the nightly-side fixes on feat-cmux-next (#25434 and #25470) don't reach it.

What changes.

  • The tick's request job checks out this workflow commit's scripts/ci/gh_budget_wait.sh (a sparse checkout at github.sha into .tick-tooling) and installs it before its gh calls.
  • Each refused gh call then:
    • waits until the reset when a budget on /rate_limit reads exhausted;
    • otherwise backs off 60, 120, 240, then 300 s, since that refusal is a limit /rate_limit does not report;
    • stops after 6 tries or one hour of waiting per job, logging every wait.
  • The job's first refusal logs GitHub's message (with its request ID) and the token's /rate_limit response headers and resources at that moment. That should tell us whether this is the secondary limit or a per-repository cap.
  • timeout-minutes goes from 15 to 75, to make room for the wait hour.
  • The script matches feat-cmux-next's copy (#25434, plus #25470's backoff), with the refusal logging added.

Unchanged: the job's concurrency group, its permissions, and how it walks commits and calls request-nightly-next.sh.

Tests. New tests/test_gh_budget_wait.py (14 tests, wired into ci-guards and test-execution.toml) drives the wrapper against a fake gh and checks the tick's wiring. These also pass: test_ci_test_execution_registry, test_ci_quota_hold_gate, test_nightly_promotion_admission, test_nightly_delivery_concurrency, test_ci_nightly_off_blacksmith, test_nightly_next_feed_watch, test_ci_workflow_permissions_are_job_scoped and test_ci_workflow_run_api_budget.

Risk: low. Only the tick changes. A waiting tick holds its job-level concurrency group longer, and newer ticks wait behind it as they do today; nothing is cancelled.

Agent session: agent session

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by CodeRabbit

  • Reliability
    • Scheduled updates now wait and retry when GitHub API rate limits temporarily block requests, helping the process continue without failing immediately.
    • The scheduled update job has a longer timeout to accommodate these waits.
  • Tests
    • Added automated checks for rate-limit handling, retries, and related scheduled workflow behavior.