ci(nightly): attest the SSH daemon assets outside the build slot, waiting out the token quota

Closed
TE
11d ago
+257-13

Summary

Nightly-next run 39368582348 (204546c) lost a 17-minute build. build-nightly-app failed at "Attest immutable SSH daemon assets" with "Failed to persist attestation: API rate limit exceeded for installation" (04:51:51Z), because a burst of repository CI had used up the run token's hourly REST quota. That step sat inside the warm build slot, so the failure discarded the build and held the slot.

What changes:

  • The attestation moves to a new job, attest-nightly-ssh-daemon, on ubuntu-24.04. It downloads the cmux-nightly-remote-daemon artifact and attests it, outside the build slot.
  • If an attempt fails, scripts/ci/wait_for_token_quota.py reads GET /rate_limit (that call doesn't count against the quota) and sleeps until the core quota resets: at least 60 s, at most about an hour. Then the job tries again, three attempts in all. Only the last attempt can fail the job.
  • publish-nightly requires the job for a build made in this run. The failure report job includes it.
  • An unsigned resume (nightly_next_unsigned_run) re-uploads the source run's assets under the same artifact name, so the job attests them again for the resume as well.
  • Nothing in the workflow verifies this attestation (publish checks the manifest with verify_remote_daemon_release.py), so attesting after the upload changes no consumer.

Known gap: a run whose build succeeded but whose three attestation attempts all failed (more than two hours without quota) does not publish. If its notarization also went pending, a later run's notarization recovery could publish those assets without an attestation. Before this change such a build failed outright.

Testing

  • tests/test_nightly_ssh_daemon_attestation.py (new, committed before the fix). On the base it gives 2 failures and 1 error; on the head it passes. It checks that the build slot attests nothing, the Linux job and its three-attempt retry chain, the quota waits, that the resume is covered, that publication requires the job, and the wait arithmetic.
  • tests/test_nightly_next_recovery.py: updated for the new job in publish's condition and the failure report; passes.
  • tests/test_nightly_unsigned_resume.py: passes.
  • actionlint: only the existing SC2129 note.
  • A review subagent found that the first version blocked publication of unsigned resumes; that is fixed in f6278f625cc.

Observed / Not run

  • Observed: the failure log of 39368582348 (the rate-limit message above). It was 1 of 25 nightly-next runs since 2026-10-19 17:20Z.
  • Not run: a nightly-next build with this change, and a real quota-exhaustion retry.

Changelog

none

šŸ¤– Generated with Claude Code

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by cubic

Nightly runs no longer lose a build when attestation hits the GitHub API rate limit. The SSH daemon attestation now runs in its own Linux job outside the warm build slot, and retries after waiting out the token quota.

  • build-nightly-app no longer attests; attest-nightly-ssh-daemon downloads cmux-nightly-remote-daemon and attests it with up to three attempts, waiting 60 seconds to about an hour between failures.
  • publish-nightly requires the attestation job for builds made in the run, and the failure report includes it; the publish contract test (test_nightly_universal_build.sh) now expects this gate.
  • Unsigned resume runs re-upload the source assets under the same name, so the job attests them again and a resume can't publish unattested assets.
  • The new attestation test is registered in the release-notary guard group and test-execution.toml.
  • The cmux-tui resolve guard exempts the attest job since it checks out only the quota-wait script at github.sha and ships no source.
  • If all three attempts fail (more than two hours without quota), the run doesn't publish; a later notarization recovery could still publish those assets without an attestation.