ci(cla-rerun): re-run CLA checks just after the budget reset; guard rejects only with API-failure evidence; cover forks

Merged
TE
25d ago
+436-124

Follow-up to #25326 (the scheduled CLA quota re-run).

What was wrong

On 2026-10-26 from about 08:40Z, CLA Assistant and CLA policy guard were red on every open pull request into main (#25403, #25406, #25407, #25408, #25411, and later #25172 and #25390). The GITHUB_TOKEN installation budget was spent. The 08:57Z sweep logged 0 of 5000 left (floor 1000) and skipped, which is correct. The gaps were these:

  • Guard rejects were re-run blindly. Every CLA policy guard failure was re-run until attempt 8, with no evidence. The guard hides every PolicyError (validate-cla-policy.rb), so a real policy rejection got 7 re-runs too.
  • Forks were never covered. GitHub lists no pull_requests on a fork's run, so current_head was always false for them (#25411).
  • Timing. The budget resets hourly (about :13 past on 2026-10-26), and push-triggered workflows drain it again within minutes. The */10 cron really fires every 25 to 40 minutes, so a pass usually landed while the budget was empty again.
  • The base was never filtered. That part was fine: main pull requests were already covered.

Change

  • One planned pass per reset. When the budget is low and the reset is under 15 minutes away, the pass sleeps until 20 s after the reset, reads rate_limit once more and runs then. This is not a loop. A farther reset is left to a later cron.
  • Its own workflow. .github/workflows/ci-cla-quota-rerun.yml replaces the cla-quota-rerun job in ci-queue-janitor.yml. It has cron */10, concurrency group ci-cla-quota-rerun with cancel-in-progress false, and a job timeout of 22 min. A pass waiting for the reset never holds the macOS sweep's concurrency group. A dispatch defaults to a dry run.
  • Floor. It drops from 20% to 10% (minimum 500). Re-runs are capped at (remaining - floor) / 40, reserving 40 GITHUB_TOKEN calls per re-run, so the pass never drains the budget itself. Right after a reset the floor never blocks.
  • Coverage. Open pull requests into main and feat-cmux-next are read per base (up to 5 pages each, with a warning past that). They are mapped by head SHA, so forks count. A same-repository run whose own pull request is into another base is left alone. The App token gets pull-requests: read.
  • A guard reject is re-run only with evidence of an API failure:
    • (a) its own log shows rate limit, HTTP 403 or 403 Forbidden before the reject line, or
    • (b) a CLA Assistant job, on any pull request, logged API rate limit exceeded for installation within 15 minutes of it. The budget is the repository's and both workflows share it. A secondary or user rate limit is not evidence.
  • Any other reject is never re-run. It gets a ::warning:: that names it as possibly a real policy rejection.
  • Case (b) is needed. The guard's own log never shows the API error, because it prints only the generic line. Every guard log from this outage looks the same.

cla.yml, cla-policy-guard.yml, validate-cla-policy.rb and the supply-chain tests are unchanged. A re-run changes no policy: the same trusted workflow runs again on the same head.

Risk

Low. The job is continue-on-error, scheduled or dry-run dispatch, and its only write is rerun-failed-jobs. Compared with before, it re-runs fewer guards (only with evidence) and more forks. It holds a Linux runner for up to 15 minutes before a near reset.

Tests

tests/test_cla_quota_rerun.py has 20 cases. They cover:

  • main and feat-cmux-next bases, with other bases skipped
  • a run whose own pull request is into another base
  • per-base reads
  • a fork matched by its head
  • a guard's own 403
  • a 403 after the reject, which is not evidence
  • a bare reject: never re-run, warned
  • installation-limit evidence inside and outside the window
  • a secondary limit, which is not evidence
  • one pass just after a near reset, a low budget after the reset, a far reset that is not waited for
  • the floor cap

They passed on cmux-ci, along with unittest discover -p 'test_ci_*.py' (2345 tests) and check_workflow_concurrency.py.

Agent session: agent session

🤖 Generated with Claude Code

Summary by CodeRabbit

  • CI Improvements
    • CLA checks retry qualifying rate-limit and API-failure errors for open pull requests targeting main or feat-cmux-next, including pull requests from forks.
    • Retries are limited to recent failures on the latest pull request revision and subject to an attempt cap. Guard rejections without API-failure evidence are not retried.
    • Reruns run through a scheduled or manually triggered workflow; manual runs include a dry-run option.
    • Reruns respect API request limits, briefly waiting for a budget reset when appropriate and adjusting retry capacity to available budget.