ci(publish): back off a secondary rate limit, upload one at a time, log the refusal

Merged
TE
11d ago
+401-49

The release publisher no longer loses a built nightly to a 10-minute burst of refused uploads. It backs off GitHub's secondary rate limit within a 30-minute budget, uploads one asset at a time, and logs exactly which limit refused it.

Why. Run 40298250989 attempt 3 (nightly-next 52d22f38) had already built, signed and verified the nightly when 18 upload POSTs were refused with "API rate limit exceeded for installation", between 12:04:51Z and 12:14Z. The refused responses had no retry-after and no x-ratelimit-* headers. At the same moment, the janitor's /rate_limit read with github.token showed core at 4899/5000 (12:05:11Z). So this is GitHub's secondary limit on write bursts, not the hourly budget: every workflow's GITHUB_TOKEN counts as one actor. The publisher waited a flat 60 s per refusal, hit its 600 s cap, and failed.

What changes:

  • publish-release-assets.py:
    • It honors retry-after and x-ratelimit-reset when present. Otherwise it backs off 60, 120 and 240 s, then 300 s per wait, inside its budget.
    • The budget comes from CMUX_RELEASE_RATE_LIMIT_SECONDS, default 600. The nightly asset publish sets 1800, and publish-nightly's timeout goes from 90 to 120 min.
    • Uploads run one at a time instead of 2 in parallel, with a 1 s pause between writes.
    • On every refusal it logs all response headers plus a /rate_limit snapshot (that read is free), so the next red names the resource. Snapshot calls can't recurse.
    • A rate-limited refusal no longer uses up one of the 3 transport attempts; the budget bounds it instead.
  • gh_budget_wait.sh (#25434): when no budget reads exhausted, it backs off the same way (60, 120, 240, 300 s) instead of a flat 60 s. It now makes up to 6 tries, still within the job's one-hour cap.

Ordering, verification, immutability and feed-last guarantees are unchanged. Tests:

  • tests/test_github_release_publication.py: 40 pass. New ones cover the backoff sequence, header and snapshot logging, and the env budget. The parallel-peak test now expects 1.
  • tests/test_gh_budget_wait.py: 16 pass, including the backoff sequence with a fake sleep.
  • Also passing: the nightly resolve, coverage, concurrency, recovery and ring tests, and the universal-build script.

Risk: normal. Serial uploads make a publish a little slower (about a dozen assets, each a few seconds to a minute), in exchange for not tripping the secondary limit ourselves.

Agent session: agent session

šŸ¤– Generated with Claude Code