ci(nightly-next): read branch refs with git ls-remote so an exhausted API budget never stalls the tick

Merged
TE
2d ago
+161-18

The NEXT nightly stalls when the repository's Actions installation token runs out of API quota.

  • nightly-next tick runs 41791114042 and 41791169626 logged could not read nightly-next (gh api repos/manaflow-ai/cmux/git/ref/heads/nightly-next); waiting for the feed watch's backstop dispatch and promoted nothing. The rolling rule reads the head through the refs API with GH_BUDGET_WAIT_ATTEMPTS: 1 (from #32328, which made the tick's housekeeping fail fast instead of waiting out a limit; the single API read of the head is from #27179). With the budget gone, that read failed and the rule waited.
  • Continue nightly notarization 41790985234 failed at preflight "Verify source run and recovery artifacts" on its first read: HTTP 403 API rate limit exceeded for installation.

The fix:

  • Branch refs come from git ls-remote https://github.com/OWNER/REPO refs/heads/X. The repository is public, so that needs no token and spends no REST budget. The refs API stays as the fallback when ls-remote fails. This covers every place the tick and its promotion path read a ref through git/ref:
    • scripts/ci/nightly_next_rolling.py: nightly-next's head (main passes watch.ls_remote_sha). With the head readable, a later failed read promotes, as the rule already said, and the promotion request step waits through the limit.
    • scripts/ci/nightly_next_stuck_run.py: Api.tip() (the unstick job).
    • scripts/ci/nightly_next_feed_watch.py: GitHub.ref_sha() (the feed watch's backstop, which the rolling rule falls back on). The shared helper ls_remote_sha lives here.
    • request-nightly-next.sh already used git ls-remote.
  • Notarization preflight: a nightly-next source run, which this workflow always turns away, now exits ready=false before any API read. The remaining reads go through scripts/ci/gh_budget_wait.sh (sparse checkout of main), which waits for the budget's reset and retries instead of failing on the first try. The new timeout-minutes: 75 bounds that wait.

No permissions, secrets or runner changes.

Tests: tests/test_nightly_next_rolling.py drops the old "no ls-remote" assertion and adds tests for: the head coming from ls-remote with no refs API read, a rate-limited refs API still deciding, and a failed ls-remote falling back to the API. tests/test_nightly_next_stuck_run.py adds the same for Api.tip(). Fleet run 41793287292: rolling 34 OK, feed watch 94 OK, stuck run 27 OK; run 41792729886: gh_budget_wait, runner set, off-Blacksmith, overflow switch and signing keychain tests OK.

Once this lands, one nightly-next tick is dispatched so a nightly carrying 9bb54041 builds.

Agent session: agent session

🤖 Generated with Claude Code

Summary by cubic

Stops the nightly-next tick from stalling when the repository's installation API budget runs out: branch refs are now read with git ls-remote, which needs no token and spends no REST budget, instead of the refs API.

  • nightly-next's head comes from git ls-remote in the rolling rule, stuck-run rescue, and feed watch, with the refs API as fallback when it fails.
  • The notarization workflow's preflight turns away nightly-next source runs before any API read and waits through rate limits with gh_budget_wait.sh, bounded by a new 75-minute timeout.
  • The notarization tests now select the preflight's check step by id instead of position to match the new steps.

Summary by CodeRabbit

  • Chores
    • Nightly build checks can read branch revisions directly and fall back to the GitHub API when needed.
    • Preflight checks now handle nightly-next source verification without querying the source run through the API.
    • The preflight job has a 75-minute timeout.
  • Tests
    • Added coverage for direct revision lookups, API fallback behavior, and nightly verification setup.