ci(cmux-next): the DEV REF capture retries a rate-limited lookup and compiles instead of reusing

Merged
TE
10d ago
+311-42

DEV run 40428066457 at c28f2f1f took no REF capture, so the fleet whole-app proof has no hybrid-ref-12 to build from. The capture-decision step's gh api lookup hit "API rate limit exceeded for installation" at 02:11:03Z and exited 1 before it wrote capture. Because the step is continue-on-error, the run stayed green. A rerun would not have fixed it: the product was a reuse hit, so the rerun would have compiled nothing.

  • hybrid_ref_capture.py have: the artifact lookup, moved to Python. It waits out a rate limit (Retry-After or x-ratelimit-reset) and backs off on 5xx and network errors, for 600 s at most, then fails with the last error. A 403 that is not a rate limit fails at once.
  • The decide step now runs before the product-reuse lookup. A key with no complete capture sets its new compile output, which SKIP_REASON reads, so that build really compiles and captures. Most landings share a key that already has a capture and still reuse.
  • An incomplete capture's manifest is now named hybrid-ref-capture-incomplete-KEY (it was hybrid-ref-capture-KEY-incomplete-RUNID, which no exact-name lookup finds). have counts the ones from the last day, and while one exists the key does not skip reuse. A key whose capture always comes out incomplete therefore forces at most one extra compile a day, not one on every landing. hq's hybrid-ref-run.sh reads only complete manifests, so the rename doesn't affect it.
  • A lookup that still fails sets capture=true compile=true, prints an ::error:: annotation and fails the step. The job is still not failed, so DEV publication is never held, but the run shows the failure and the build compiles and captures.
  • The decide step's timeout goes from 1 to 12 minutes. A capture still starts only before job minute 40, unchanged.
  • No new token: this job has no App token, and minting one here would be a secrets change. It stays on github.token.

Not touched: the nightly's capture in #25812, which has the same gh api lookup and can switch to have after this lands.

Tests

python3 tests/test_cmux_next_hybrid_ref_capture.py: 9 red at base cdc41df7824 (first commit); 20 of 20 pass at the head, eight runs in a row. test_the_decision runs the decide step's real shell on four have answers. tests/test_dev_release_product_reuse.py passes. One of the 9, test_off_by_default_and_never_in_the_way, was already red on base: the job's timeout-minutes became an expression. It now reads the mini branch, 60. actionlint gives the same 4 findings as base, all pre-existing. prepush-gate is clean.

Observed / Not run

Ran: the test file above (base and head), actionlint and prepush-gate. Read the 40428066457 DEV compile job log (job 121939218995) for the failure.
Not run: a real DEV daily with capture on. That is the next step: a DEV daily dispatched at the tip after this lands.

Changelog

none

šŸ¤– Generated with Claude Code

Agent session: agent session