ci(nightly-next): retention prune for the NEXT downloads on R2 (dry run first)

Merged
TE
10d ago
+468-1

This is the R2 retention follow-up named in #25656, which took the cmux NEXT feed off GitHub release uploads. Leo, on the 13-18 minute upload step: "How can you possibly be uploading for anything more than one minute? ... You find that shit. You unbottleneck it."

Since #25656, publish-nightly and republish-nightly-deltas put every nightly-next DMG and delta on R2 under cmux-nightly-next/nightly-next/next/, about 240 MB a build. Until now nothing removed them.

What it does

A new last step of republish-nightly-deltas, "Prune old nightly-next downloads on R2", runs scripts/ci/nightly_next_r2_prune.py with the track's existing scoped NIGHTLY_NEXT_R2_* token, in the same release-next environment.

Why it runs there: that job holds release-publish-<tag>. publish-nightly and republish-nightly-deltas, the only writers of this feed in nightly.yml, hold it, so no publish-path feed write runs while it deletes. nightly-next-rollback.yml writes outside that group, but it only removes items, and --to serves an item already on the feed, so a rollback never makes a deleted file live.

Candidates: only cmux-nightly-next-macos-<arch>-<build>.dmg and ...-<build>-<base>.delta directly under the prefix. Feeds, notes, verdicts, dSYMs and anything in a subfolder are never touched. The prefix is fixed to nightly-next/next/, and every --feed has to be an appcast*.xml directly under it.

Kept:

  • every file the live feeds name, read from the bucket rather than the cache, and read again before each delete;
  • the newest 20 builds, where a delta counts toward its target build;
  • the delta bases a later build may name: the two newest builds at least 24 and at least 72 hours old, the same ages as generate-nightly-deltas.sh, which a test checks;
  • anything under 7 days old.

Fails closed. It deletes nothing when a feed is missing, unreadable or has no items, or when the listing is incomplete. If a feed read or a delete fails partway through, it stops and reports what it already deleted. A run deletes at most 100 objects.

Dry run by default. It deletes only when the repository variable NIGHTLY_NEXT_R2_PRUNE_DELETE is 1. Turn that on after a dry-run summary looks right.

Never in the way. The step has continue-on-error and a 3-minute timeout. It runs after the revised feed is live, so it adds nothing to the feed's critical path.

Tests

  • tests/test_nightly_next_r2_prune.py checks that it:
    • keeps what the feed names, the newest builds, the delta bases and new objects;
    • never touches anything outside the prefix, or anything that isn't a DMG or delta;
    • deletes nothing on a dry run;
    • fails closed;
    • reads the feed before every delete;
    • stops on a failed read;
    • refuses unsafe settings and feed keys.
  • The nightly step-coverage, concurrency, secrets, republish-guard and identity tests all pass.

Risk

Low. It's a dry run until the variable is set. In delete mode it only removes old builds that no feed names and that aren't recent or likely delta bases, only under the track's own prefix, and only with the existing scoped token. No secrets, environments or permissions change.

Agent session: agent session

šŸ¤– Generated with Claude Code