#25706 ci(dev tip): publish the full DMG first, add the deltas after

Leo, 2026-10-26: "And I thought we were incremental... Nobody has come back with the incremental building results."

Follows #25673 (the tip DMG builds while verify runs). This is move 2 of the incremental-build plan: publish the full DMG first, add the deltas after, the way nightly does with generate-nightly-deltas and republish-nightly-deltas.

What changes

• tip-dmg no longer fetches older DMGs and no longer runs Sparkle's BinaryDelta. Its appcast ("Generate the full-DMG tip appcast") offers the notarized, stapled full DMG alone. The step fails if any .delta file or <sparkle:deltas> element shows up, so the first feed write never names a delta.
• tip writes that feed with the same compare-and-swap as before. It now outputs written=true and the run_tag of the DMG name the feed serves.
• tip-deltas ("Add deltas to the cmux NEXT DEV tip feed", macOS, needs: [sign, tip-dmg, tip], runs only when needs.tip.outputs.written == 'true'):
  1. It reads the feed from R2 directly, not through the CDN. tip_feed.py bases returns the delta bases only when all of these hold: this build's item is the feed's newest, the item is served from tip's DMG URL, and the item has no deltas yet. Otherwise it prints "skip" and the job adds nothing.
  2. It fetches those older DMGs and runs sparkle_generate_appcast.sh against the same DMG bytes, under the same name.
  3. In a compare-and-swap loop it reads the feed from R2 and runs tip_feed.py add-deltas. That adds the generated item's <sparkle:deltas> to the live item for this build and leaves every other byte of the live feed as it is. It refuses (exit 4, no write) if the build is no longer the newest, the item is gone, the item already has deltas, or it is another attempt's DMG. It fails if the generated full enclosure (URL, length, edSignature) differs from the live one, or if a delta URL names a file this run did not make.
  4. On the first pass it uploads the deltas write-once. Then it does r2_cas.py put --if-match <etag>. It never uses --if-absent, so a missing feed is never created here.

How this meets hotswap-opus's checklist

Every feed write is a compare-and-swap, run by tip_feed.py against the live feed with If-Match. Neither write is a blind put:

• tip: merge, as before.
• tip-deltas: add-deltas, plus --if-match on the put.

The feed only names objects already uploaded. The DMG is uploaded before tip's write. The deltas are uploaded before tip-deltas's write, and add-deltas checks that each named delta is a file this run has.

The second write drops no newer build. It is built from the latest R2 read, with only this item's deltas added. A test checks that the merged feed, with the added deltas removed, is byte-identical to the live feed, and a round trip against the real live feed reproduces it exactly. If a newer build lands between the read and the put, the put gets a 412. The job then reads again, add-deltas says there is nothing to add, it writes nothing, and it removes its unused deltas.

Nothing is added to an item that is no longer in the feed. bases and add-deltas both refuse in that case. They also refuse when the item is not the newest, when it is from another attempt, or when it already has deltas.

Delta names are unique per attempt and write-once. They use SPARKLE_DELTA_NAME_PREFIX: cmux-NEXT-DEV-tip-<run_id>-<tip-deltas run_attempt>- with --write-once, and a re-upload of identical bytes is accepted.

The cleanup trap's rule holds.

• After a feed write with no clear answer (unsure), the deltas stay.
• After a clear refusal, or a refusal from add-deltas, this attempt's deltas are removed.
• A failed upload writes no feed.

If the delta step fails after the full-DMG write, the feed stays exactly as tip wrote it: valid, signed, offering the full DMG with no deltas for this build. Sparkle installs the full DMG. The next build's tip-deltas makes deltas from this build's DMG, as before.

The jobs stay red-but-advisory. tip-dmg, tip and tip-deltas each keep job-level continue-on-error: true.

No concurrency group. tests/test_nightly_delivery_concurrency.py now lists tip-deltas as well.

Timing. Sign done → DMG in the feed, before and after, comes from the same harness on real runs. It is reported after landing.

Idempotency

• Appcast generation: Ed25519 signatures are deterministic, so regenerating against the same DMG bytes gives the same full-enclosure signature, and add-deltas requires that.
• R2 uploads: the uploads are write-once, and an identical re-upload is accepted. A re-run of tip-deltas makes new delta names. Its bases step skips if an earlier attempt already added deltas.

Risk

Medium, limited to the DEV tip feed. publish is unchanged.

• Worst case if tip-deltas breaks: the tip feed has no deltas, so updaters download the full DMG of about 159 MB instead of a delta of about 13 MB.
• Worst case if the new tip-dmg appcast breaks: the tip feed stops advancing, as with any tip failure. Restore is a revert, within the hour.
• Cost: tip-deltas is one more Blacksmith macOS job of about 4 to 5 min per fresh build. The macOS minutes for deltas are unchanged; only the job start is added.

Signals

• Success: Publish the cmux NEXT DEV tip feed finishes about 3.5 min earlier relative to sign. Then Add deltas to the cmux NEXT DEV tip feed turns green, and the live feed's newest item gains <sparkle:deltas> a few minutes later.
• Failure: the tip feed's newest item never gains deltas while tip-deltas is red. If tip itself fails in "Require the archive verify tested" or in the R2 upload, revert.

Tests

The following pass locally, and actionlint is clean:

• python3 tests/test_cmux_next_dev_tip_feed.py
• tests/test_nightly_delivery_concurrency.py
• tests/test_ci_signing_keychain_is_deleted_before_bundle_code.py

New tests:

• bases and add-deltas unit cases: a newer build, a gone build, another attempt, deltas already added, a different signature, a delta that was not uploaded, and the byte-for-byte check.
• The tip-deltas write step run in bash against stubbed R2, covering: the normal write, a newer build arriving mid-write, a lost race that is retried with a single upload, a build that is not the newest, a missing feed, a pending write, and a failed upload.

Agent session: agent session

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by CodeRabbit

• Improvements
  • Tip builds now publish the full DMG before adding delta updates to the appcast, so the complete installer is available first.
  • Delta updates are added only when the published build remains the newest eligible build. Feed changes are retried after conflicts to avoid overwriting newer updates.