#26922 ci(release): verify every stable Sparkle feed delivers an update

Fixes https://github.com/manaflow-ai/cmux/issues/26729

What broke

releases/latest was v0.65.2.2, published with 0 assets. All 12 signed assets sat on a hidden staged draft with the same name (release id 425503700, untagged-7319b3bc503c80f29055). So every Sparkle feed URL an installed app reads 404ed:

• releases/latest/download/appcast.xml and appcast-<arch>.xml
• the R2 feeds at files.cmux.com/stable/, which pointed at v0.65.2.2 DMGs that were not there

Every 0.65.2 update check showed "error retrieving update information". The release run was green.

The cause: v0.65.2.2 was tagged from release/v0.65.2.2, a hotfix branch cut before https://github.com/manaflow-ai/cmux/pull/21140. A tag runs the workflow file at that tag, so the old softprops/action-gh-release publish-by-tag step created a second release and published it empty, as happened with v0.65.2.

Repair (done, no deletion)

I uploaded the draft's 12 assets onto the published release 425504204. Before uploading I checked:

• Each sha256 matches the draft's GitHub digest.
• Every enclosure in the three appcasts verifies against the app's SUPublicEDKey (EdDSA), with the declared length.
• The three DMGs pass stapler validate and spctl.
• The draft's appcasts are byte-identical to the live R2 feeds.

After the upload all six stable feeds and every DMG they list return 200 with the right size. Nothing was deleted: the tag, the published release and the leftover draft are untouched.

Guard (this PR)

• scripts/ci/verify_stable_feed.py loads all six default stable feeds (GitHub latest and R2), requires signed enclosures, and requests one byte from every enclosure. It reads that byte and compares the declared length with the full size reported by the server. If no full size is reported, it skips the length comparison. It does not download complete DMGs or cryptographically verify signatures.
• release.yml runs a separate verify-stable-feed job after publication, adding the release's own tagged feeds. Non-latest backports check only their tagged feeds. Homebrew remains gated on build-sign-notarize.
• stable-feed-watch.yml checks the default feeds hourly from main, regardless of which branch published them. It does not inventory non-latest backport tags; those tagged feeds are checked only by release workflows that include the guard.
• Failing feeds retry in rounds within one shared settle window, and passing feeds leave subsequent rounds. The deadline limits new feed-check retries; requests already running may finish afterward under the workflow's job timeout.
• tests/test_verify_stable_feed.py runs 24 behavioral tests through r2-upload-tests.yml, including an actual HTTP response that closes before sending its advertised byte, malformed enclosure lengths, retry fairness, and deadline handling.

Related: https://github.com/manaflow-ai/cmux/pull/26611 adds finalize-release.py adopt for an empty published release. The asset repair above was completed during the original investigation; this takeover made no release-asset changes.

Validation

Observed / Not run

Observed on 02327be2a3ffb64308aacf9e27d5239559b97ea5, after merging main aa4dcd453ce9a2ce5baa66c0863ef8d5758a4350:

• python3 tests/test_verify_stable_feed.py: regression commit 8ad136ddd5fcd3169899909829106b5dc0919cc0 fails with 3 assertion failures and 1 conversion error on the pre-fix checker. The same command passes all 24 tests after the fix. The independent review also reran all 24 successfully.
• python3 scripts/ci/verify_stable_feed.py --tag v0.65.2.2 --attempts 1: all nine live feed checks pass, including their enclosure probes.
• All three workflow YAML files parse; the release guard's shell passes bash -n; git diff --check passes.
• python3 scripts/ci/validate_test_execution_registry.py --base-sha origin/main: passes (509 tests; an existing unrelated registration warning remains).
• python3 scripts/verify-local.py --only project-tests: 16 tests pass on the merged tree. The repair does not change project normalization.
• Independent correctness review passes the final repair head.

Not run: a release publication workflow, a complete DMG download/signature verification, or an end-to-end Sparkle update from 0.65.2. This changes CI tooling only, so no native build or runtime launch is needed. Localization audit: no product UI, help, or localized resources changed.

Changelog

none

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Guards every stable Sparkle feed so a broken release can never again ship behind a green CI run. When v0.65.2.2 was published with 0 assets, every update check 404ed; this adds scripts/ci/verify_stable_feed.py, which loads each stable feed (GitHub latest/download and R2) the way Sparkle does and verifies every enclosure downloads with its declared length. It runs in a new verify-stable-feed release job and hourly via stable-feed-watch.yml from main, which also covers tags cut from older branches that run their branch's workflow.

• The check never downloads DMGs; it uses a one-byte range request and fails if the probe body is missing. A 404 is not retried, other errors are, and --settle-seconds retries failing feeds in rounds within a shared window to tolerate GitHub's cached 404 right after an upload.
• Backport tags that are not latest verify only their own feeds, and 24 unit tests run in r2-upload-tests.yml and the execution registry.

Summary by CodeRabbit

• Chores
  • Stable release feeds and their downloadable enclosures are checked automatically on a schedule and during tagged releases, with an option to run scheduled checks manually.
  • Feed checks validate enclosure links, declared file sizes, and signatures. Temporary failures are retried before an error is reported.
  • Relevant changes also trigger feed-check tests in pull requests.
• Tests
  • Added automated coverage for feed validation, retries, release-specific checks, enclosure details, and error handling.