ci(dev tip): build, sign and notarize the tip DMG while verify runs

Merged
TE
10d ago
+267-45

Leo, 2026-10-26: "And I thought we were incremental... Nobody has come back with the incremental building results."

What changes

In cmux-next-dev-daily.yml, the tip job waited for the Verify job before it did any DMG work. Verify's queue plus run added a median 3.6 min from "sign done" to the start of the DMG work (13 recent publish runs).

The job is now two jobs:

  • tip-dmg ("Build the cmux NEXT DEV tip DMG", macOS) needs only sign, so it runs while verify does. It reads the live feed (a stale build stops here), builds the DMG, signs it, notarizes and staples it, checks the key and build, and generates the appcast with deltas. Then it uploads the DMG, deltas, appcast and the digest of the archive it built from as an artifact for this attempt. It holds no R2 key and writes nothing public.
  • tip ("Publish the cmux NEXT DEV tip feed", now Linux) needs [sign, verify, tip-dmg]. Like publish, its if requires needs.verify.result == 'success' and needs.verify.outputs.verified == 'true'. It also requires needs.tip-dmg.outputs.ready == 'true', which only the step after the artifact upload sets. Before any upload it checks that the digest tip-dmg built from equals tested.sha256, the digest verify passed, byte for byte. The R2 upload step is the same as before: write-once archives first, then the compare-and-swap feed write.

Why the feed is gated rather than rolled back. You asked for one of two designs: tip publishes only after verify passes and does its work in parallel, or a failed verify rolls the feed back. A rollback would leave a bad build in the feed until the revert lands, and installed apps could pull it in that window. A gate means a build verify rejects never reaches the feed, so there is nothing to roll back. All a failed verify costs is one notarized DMG that nobody can see.

How this meets hotswap-opus's checklist

  1. codesign in tip-dmg. tip-dmg runs Apple's /usr/bin/codesign --verify --deep --strict --verbose=2 on the Linux-signed app before hdiutil create, the DMG codesign --force and notarytool submit. The step runs under set -euo pipefail, so codesign's exit status survives the tee. It has no continue-on-error and no || true. If the output contains "sealed resource is missing or invalid", the step fails before the DMG exists. Notarization and stapling still come before the appcast that names the DMG. The SUPublicEDKey and CFBundleVersion guards are unchanged. publish keeps needs: [sign, verify] and its gate. Tests assert all of this, both with string checks and with YAML-level checks.
  2. Feed writes. Unchanged in this PR: the only feed write is still the tip_feed.py merge compare-and-swap with If-Match, after the write-once uploads, and the cleanup trap rule is the same. The upload step's script is byte-identical apart from losing its if:, and the existing bash harness for that step still passes. TIP_RUN_TAG comes from tip-dmg's outputs, so a re-run of tip alone uploads exactly the files tip-dmg signed and named. A re-upload of an identical object is accepted by --write-once. Each tip-dmg attempt hands off under its own artifact name (cmux-next-dev-tip-<attempt>).
  3. No concurrency group on publish, tip-dmg or tip. tests/test_nightly_delivery_concurrency.py now lists tip-dmg too. test_cmux_next_dev_tip_feed.py checks the whole file.
  4. Timing. Before/after numbers come from the same runs, measured as sign done → DMG in the feed. They are in the comment below and will be filled in from real runs after landing.

Both tip jobs keep job-level continue-on-error: true, so they stay red-but-advisory and the dispatcher never re-publishes a source run because of them.

Risk

Medium, confined to the DEV tip feed. The rolling prerelease (publish) does not change. This workflow only runs from main (sign requires refs/heads/main), so the first real check is the first DEV publish after landing.

  • If it breaks, the tip feed stops advancing. Installed apps keep the current build; nothing bad reaches them. Restore is a revert, within the hour.
  • Extra cost when verify fails: tip-dmg's notarization runs anyway, because it no longer waits. That is Blacksmith macOS minutes only, with no effect on the feed.
  • Owned-pool rescue: if the rescue cancels a run whose verify is stuck on an owned Mac, the cancel now also stops tip-dmg's work in progress. The re-run redoes it under new attempt names.

Signals

  • Success: on a DEV publish run, Build the cmux NEXT DEV tip DMG starts right after Sign cmux NEXT DEV, alongside Verify signed cmux NEXT DEV. Publish the cmux NEXT DEV tip feed then finishes about 0.5 min after the later of the two. measure_next_build.py --per-landing shows l>tip dropping by verify's share.
  • Failure: tip is skipped while verify succeeded and tip-dmg is green, or tip fails in "Require the archive verify tested". In either case, revert.

Tests

python3 tests/test_cmux_next_dev_tip_feed.py, tests/test_nightly_delivery_concurrency.py, tests/test_ci_signing_keychain_is_deleted_before_bundle_code.py and the tests/test_ci_*workflow* guards pass locally. actionlint is clean. The new tests run the digest gate in bash: a matching digest places the DMG, appcast and deltas where the upload step reads them, a different digest publishes nothing, and an empty run tag publishes nothing.

Agent session: agent session

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Reliability
    • Daily preview apps are checked for valid signatures before installers are built, signed, and notarized.
    • The download feed is updated only after release verification succeeds and the installer is confirmed to match the tested app.
    • Recovery from interrupted or repeated release attempts is safer: valid downloads are retained when a feed update’s outcome is uncertain, while uploads from unsuccessful attempts are cleaned up when appropriate.