ci(nightly-next-tick): unstick reads as the cmux-bot App; fewer request-job reads

Merged
TE
2d ago
+425-16

The repository's workflow GITHUB_TOKEN pool (5,000 calls an hour, shared by every workflow) ran dry every hour on 2026-11-04. The token-budget sample (14:01-15:01Z) put nightly-next-tick first among single spenders: the unstick job made about 8.6 REST calls per job and the request job about 6.9, at about 67 jobs an hour each.

Stacked on #35062 (git ls-remote for branch refs). Its two commits show in this diff until it lands; this one merges after it, with main merged in.

What changes

  • unstick reads go to the cmux-bot App. The job mints a cmux-bot installation token with the credentials the repository already stores (vars.CMUX_BOT_APP_ID, secrets.CMUX_BOT_APP_KEY, the same pattern as ci-queue-janitor and ci-owned-pool-rescue), scoped to actions: read and contents: read. It then installs scripts/ci/gh_app_token.sh in a new --reads-only mode:

    • a gh call that only reads goes out as the App: a gh api GET with no request fields, a GraphQL query, or gh run|workflow|pr|issue|release list|view;
    • when the App is refused, the same read is sent again with github.token;
    • every other call (the cancels, force-cancels and re-runs, and anything the wrapper cannot classify) runs once as github.token, untouched. GH_APP_TOKEN_ATTEMPTS=1 sits next to the existing GH_BUDGET_WAIT_ATTEMPTS=1, so neither wrapper adds a hidden backoff;
    • with no App token (a failed mint, or no variable), nothing is installed and the job runs exactly as before.

    Without --reads-only, gh_app_token.sh behaves as before (the nightly publish path).

  • The unstick job's runner is bounded to the linux_2 pool's members. A job that holds an App key may not run where a variable can move it (tests/test_ci_fork_runner_routing.py). Its runs-on now names blacksmith-2vcpu-ubuntu-2404 and blacksmith-4vcpu-ubuntu-2404 (the members in .github/runners.json). That matches every value the RUNNER_POOLS reconciler writes, and today's value is blacksmith-4vcpu-ubuntu-2404.

  • The request job keeps only its GITHUB_TOKEN. CI_CONTROL_RUNNER may move it to a reused machine, so by design it holds no secret and no App token (tests/test_ci_cloud_overflow_switch.py). Its routing is unchanged. It spends fewer reads instead:

    • the rolling rule reads the gap from nightly-next's head to the tip from the tick's own 150-commit checkout of feat-cmux-next (git_oldest_after), not the compare API. The compare API is still the fallback when the head is not in the checkout (a train commit, or a head past the shallow history);
    • when the tip is already on nightly-next (at or behind its head), the rule now reads nothing more and returns go=false. Before, it listed the runs and the promotions, then requested, and the request walk stopped at the tip with "already on nightly-next". No commit that could be promoted is skipped.

  • Neither job starts for a cmux-next or cmux-tui artifacts run on nightly-next. A promotion's push starts those runs (about 7 an hour in the sample). They make no feat-cmux-next commit buildable and are not a finished nightly build. Landings, finished Nightly macOS build runs, dispatches and the 5-minute cron all start the tick as before, so no promotion waits longer.

All writes stay on the workflow token: the promotion and backfill dispatches (main's promote job admits the tick's request by its actor), cancels and re-runs.

Performance

before (main, measured)after (estimate)
GITHUB_TOKEN REST calls/h, unstick~576 (8.6 x 67)~0 (writes only: cancels and re-runs, a few a day, plus reads the App refused)
GITHUB_TOKEN REST calls/h, request~462 (6.9 x 67)~295 or less ((6.9 - 1 ref read [#35062] - 1 compare) x ~60 jobs), lower still by the ticks that find the tip already promoted (not measured)
GITHUB_TOKEN REST calls/h, tick total~1,040~295 (estimate)
cmux-bot App calls/h0~515 ((8.6 - 1) x ~60 jobs, plus about 1 token revoke per job) (estimate)
tick jobs/h, each job~67~60 (7 nightly-next cmux-tui/cmux-next completions no longer start one)

cmux-bot's installation budget had about 6k of 12.5k used, so the ~515 calls/h fit.

Tests

  • tests/test_gh_app_token.py: --reads-only sends reads as the App and every write once as github.token (a refused write is not retried and nothing waits). It falls back to github.token for a refused read without waiting. The mode is off without the flag. A wiring test checks that unstick mints with CMUX_BOT_APP_ID/CMUX_BOT_APP_KEY at actions: read, contents: read, installs before the rate-limit wait, and keeps every GH_TOKEN step on github.token with single-shot attempts. It also checks that the request job holds no App key.
  • tests/test_nightly_next_rolling.py: git_oldest_after is tested against a real git history (the oldest commit after the head and its landing time; nothing waits at or behind the head; a diverged head is unknown; a head outside the checkout falls back). When the checkout answers, evaluate reads no compare. A tip already on nightly-next reads nothing.
  • Ran: test_gh_app_token, test_gh_budget_wait, test_nightly_next_rolling, test_nightly_next_stuck_run, test_nightly_next_cancelled_tip, test_ci_cloud_overflow_switch, test_ci_fork_runner_routing, test_ci_runner_set, test_ci_workflow_run_api_budget, test_ci_nightly_off_blacksmith, test_ci_pr_runner_pool, and the workflow-wide test files; actionlint is clean.

Follow-up, not here: the request job's remaining reads (two run listings and one jobs read in the rolling rule, plus request-nightly-next.sh's per-commit reads) could move to one GraphQL query, or to an App token once the request job runs on an ephemeral label.

Agent session: agent session

🤖 Generated with Claude Code

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

Summary by cubic

Cuts the nightly-next-tick workflow's GITHUB_TOKEN usage by about 70% (from ~1,040 to ~295 REST calls per hour) so it no longer exhausts the shared 5,000-calls-per-hour workflow budget, which ran dry hourly on 2026-11-04.

Changes

  • The unstick job's reads now go out as a cmux-bot App token through a new --reads-only mode of gh_app_token.sh; its writes (cancels, re-runs) still use github.token, and refused reads fall back to it.
  • The unstick job's runner is bounded to the linux_2 pool's named members, since a job holding an App key must not run where a variable could move it.
  • The request job gets the nightly-next head from git ls-remote (no API budget) and the gap to the tip from the tick's own checkout, picked by committer date as the compare API does; the compare API is the fallback only when the head is not in the checkout or a shallow cut misses part of the range. A tip already on nightly-next reads nothing at all.
  • Both jobs no longer start for cmux-next or cmux-tui artifacts runs pushed on nightly-next (a promotion's push), which build nothing and aren't finished builds.
  • The notarization preflight turns nightly-next sources away before any API read and waits through rate limits with gh_budget_wait.sh.

All writes stay on the workflow token, so the promotion's actor check is unaffected.

Summary by CodeRabbit

  • Chores
    • Refined nightly-next workflow eligibility so only the named macOS build is excluded from the specified push events.
    • Restricted the unstick job to designated Linux runners, with an existing fallback.
    • Recognized read-only operations can use an App token when available; writes continue using the standard GitHub token.
    • Nightly rolling checks use local commit history when available, fall back to the compare API when needed, and avoid extra lookups when no gap exists.
  • Documentation
    • Clarified how nightly rolling checks look up the current head and commit gap.
  • Tests
    • Added coverage for token handling, workflow configuration, and local-history checks.