ci(nightly-next): one-command rollback of the cmux NEXT feed

Merged
TE
18d ago
+671-6

A bad cmux NEXT nightly reaches every install within one update check, and there was no quick way to take it off the feed. This adds one:

gh workflow run nightly-next-rollback.yml -R manaflow-ai/cmux --ref nightly-next -f reason="what is broken"

That takes the newest build off appcast-arm64.xml, so the feed serves the previous build again. Its signed DMG is still on the nightly-next release. -f bad_build=N pulls N and everything newer. -f to_build=N serves N again. -f dry_run=true only prints the plan.

What a pull can't do: Sparkle offers only a build above the running one (AppcastSelector uses SUStandardVersionComparator). A pull stops the bad build from spreading to new installs and to Macs that haven't taken it. Macs already on it stay there until a newer build ships, and the fix-forward revert gets that newer build. Republishing the old bytes at a higher version would mean re-signing and re-notarizing them, because CFBundleVersion is inside the signature. That waits on Apple just as long as a fixed build does. The app's local updates.rollback can bring one Mac back to the build it kept, but a Release CLI can't reach it yet; that's a follow-up.

How it works:

  • scripts/ci/nightly_next_feed_rollback.py plan drops the pulled <item> blocks and copies the rest byte for byte. It refuses to empty the feed, to target a build that isn't on the feed, or to read non-numeric versions.
  • The job runs in release-next, which deploys only to branch nightly-next, so this file reaches it with the next promotion. The job:
    • runs nightly-next-guard.py on the appcast and the R2 key, as publish-nightly does;
    • checks that the fallback DMG downloads;
    • writes the feed only while the R2 object still holds the bytes it planned from. upload-r2-object.py gets a new --replace-sha256 flag: a GET, a hash check, then a PUT with If-Match on the ETag. A publication that lands in between is therefore never overwritten, and the job asks to be run again;
    • reads the feed back after the write.
  • The delta lane (republish-nightly-deltas) used to skip its rewrite only when the live feed served a newer build. After a pull the feed serves an older one, so the lane's two-item appcast would have put the pulled build back. It now skips whenever the feed serves any build other than its own. The new cases in tests/test_nightly_republish_delta_guard.py fail on the old guard and pass on the new one.
  • The docs in docs/ci/nightly-next-canary.md add a short map of how a nightly is built, signed, notarized and published, and what each hold variable does.

A copy of this file has to be on main as well before GitHub lists it for dispatch. That follows in a main PR together with the post-publish canary.

Testing:

  • python3 tests/test_r2_upload_requests.py: 16 tests pass, including the If-Match write, a changed object, a 412 response, and the flag being exclusive with --write-once.
  • python3 tests/test_nightly_next_feed_rollback.py: 11 tests pass, including one that runs the pulled feed through nightly-next-guard.py appcast. It is wired into ci-guards (release-notary).
  • tests/test_ci_production_secrets_protected_env.py now requires release-next for the job, and passes.
  • actionlint is clean on the new workflow.
  • Not run: the workflow itself. It can't run until nightly-next has the file. The first live check should be -f dry_run=true.

Changelog

none

🤖 Generated with Claude Code