ci(main): rate-limit-safe publisher on main; auto-resume's publish jobs wait through rate limits

Merged
TE
11d ago
+566-36

Why

On feat-cmux-next, #25470 made the nightly publish path wait through GitHub API rate limits. On 2026-10-26 the installation limit refused every upload from 15:55 to 16:05Z, returning only x-github-request-id with no x-ratelimit-* and no retry-after. auto-resume-nightly-notarization.yml and release.yml check out main and run main's scripts/ci/publish-release-assets.py, so they still had the old publisher: a 600 s budget, flat 60 s waits and parallel uploads. auto-resume's publish jobs also had no gh wrapper.

Change

  • scripts/ci/publish-release-assets.py and tests/test_github_release_publication.py are now feat-cmux-next's versions, from #25470. Neither file had changed on main since the branches split. On a refusal, the publisher waits for Retry-After if it's sent. Otherwise it waits until the reset when the budget reads spent, or else backs off 60/120/240/300 s, all inside CMUX_RELEASE_RATE_LIMIT_SECONDS (default 600). Uploads are serial with a 1 s pause between writes. It logs the headers of each refusal plus a /rate_limit snapshot. A rate-limited digest download also backs off. All of this shares one per-job ledger with gh_budget_wait.sh, which fails closed.
  • scripts/ci/gh_budget_wait.sh: feat-cmux-next's ledger fixes, keeping main's once-per-job refusal logging from #25484:
    • fail closed on a bad ledger;
    • reserve a wait before sleeping;
    • never wait when the reservation can't be recorded.
  • auto-resume-nightly-notarization.yml: the publish and republish-deltas jobs install the wrapper right after their checkout of main, and give the publisher CMUX_RELEASE_RATE_LIMIT_SECONDS=1800. Neither job sets a timeout, so both keep the default 360 minutes.
  • Tests:
    • The secondary-limit and 429 wrapper tests now assert success, two attempts and the recorded wait (Astra's note on #25484).
    • Three ledger tests come over from feat-cmux-next.
    • AutoResumeTests checks the auto-resume wiring.
  • ci-owned-pool-rescue.yml: the header says the rescue grace is 40 minutes, which it is since #25510.

Risk

Medium.

  • The classic nightly (nightly.yml on main) and release.yml now run the new publisher with its default 600 s budget. Their uploads become serial, which is slower per run but not by much. They wait through a refusal instead of failing after 600 s of flat waits.
  • No timers or loops beyond bounded, logged sleeps. Every wait is capped by the publisher's budget and the job's one-hour ledger.

Overlap: #25490 and #25491 (release-App-first fallback, inert tooling) come from a helper in this same lane and touch the uploaders, not these files' logic.

Tests: tests/test_gh_budget_wait.py, tests/test_github_release_publication.py, tests/test_nightly_resume_tracks.py, tests/test_nightly_publication_marker.py, tests/test_ci_owned_pool_rescue.py, tests/test_ci_workflow_guards_are_wired.py, tests/test_ci_test_execution_registry.py, bash tests/test_nightly_universal_build.sh.

Agent session: agent session

šŸ¤– Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by CodeRabbit

  • Bug Fixes
    • Release asset publishing now handles GitHub rate limits more reliably, waiting and retrying when appropriate while respecting a shared wait limit.
    • Asset uploads are processed sequentially, and rate-limited downloads used for verification can be retried.
    • Nightly release publishing allows additional time for rate-limit waits before resuming.
    • Publishing errors provide clearer diagnostic details while keeping sensitive token information redacted.