ci(rescue): wait through GitHub rate limits instead of leaving a cancelled run un-re-run

Merged
TE
11d ago
+445-14

What broke

At 14:36:06Z the owned-pool rescue cancelled nightly-next run 40340830337 (e22f9384, the nightly-next tip). It did that to move build-nightly-app off glaeda-side-std-xcode-26.6, where no runner had picked the job up. The re-run that should follow the cancel was refused with a rate-limit 403. The watch logged gave up: HTTP Error 403: Forbidden at 14:37:38Z (rescue run 40340838656). The run stayed cancelled, and the tick won't request it again ("already on nightly-next"), so the tip never built.

Change

scripts/ci/owned_pool_rescue.py: when GitHub refuses a GITHUB_TOKEN request (read or write) for a rate limit, the rescue waits and sends the same request again instead of raising. The wait is the first of these that applies:

  • Retry-After;
  • the reset time, when x-ratelimit-remaining is 0;
  • otherwise the secondary-limit backoff: 60, 120, 240, then 300 s.

Other rules:

  • One job waits at most 15 minutes in all (RATE_LIMIT_WAIT_SECONDS).
  • A permission 403 (no rate-limit signal) still raises.
  • An App-token read that hits the rate limit falls back to GITHUB_TOKEN as before, then waits there.
  • Each wait logs the refusal's rate-limit headers, request id and message, which gives us the first full refusal headers.

ci-owned-pool-rescue.yml: timeout goes from 340 to 355 minutes to make room for the waits. JOB_TIMEOUT_SECONDS and the timeout test include them.

A rescue can no longer leave a nightly-next tip cancelled

  • Each watch thread has its own wait budget. The sweeper shares one client across threads, so without this one run's waits could spend another's.
  • Waits through a rate limit don't count toward the cancel's 20-minute settle window, so a waited-out force-cancel can't end the window before the re-run.
  • A nightly watch that gives up now logs an ::error title=nightly rescue:: annotation.
  • scripts/ci/nightly_next_cancelled_tip.sh covers the tick's blind spot. The tick never requests a commit that is already on nightly-next ("already on nightly-next"), so a cancelled tip stayed cancelled. nightly-next-tick.yml now runs this script from its own tooling checkout, before the promotion and with continue-on-error. When the tip's newest nightly run ended cancelled and nothing for that tip is still running, the script re-runs that run's failed and cancelled jobs (attempt 2 and later take Blacksmith), up to 3 attempts. Tests: tests/test_nightly_next_cancelled_tip.py.

Risk

Medium. A rescue that hits a rate limit now takes up to 15 minutes longer instead of giving up, and nothing else changes. Timers: each wait is one bounded sleep, capped per job; no new loops.

Tests: tests/test_ci_owned_pool_rescue.py covers re-run after secondary refusals and Retry-After, the reset wait, permission 403s not waiting, the bound, and the App-token fallback. 178 tests pass, as do the workflow_run_api_budget, owned_pool_sources and queue_janitor suites.

The tip run was re-run by hand (failed jobs, attempt 2).
Agent session: agent session

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes

    • CI rescue runs retry eligible GitHub API requests when rate-limited, within a capped wait period.
    • The rescue job allows additional time for rate-limit waits and cancellation settling.
    • Nightly rescue give-up messages appear as error annotations in CI.

  • New Features

    • The nightly workflow can rerun failed or cancelled jobs from the latest cancelled run, up to the configured attempt limit. It skips runs that are still active, have no run data, or have reached that limit.