ci(stable-next): cmux NEXT Stable, a nightly-picked, gated sibling of the cmux NEXT nightly

Merged
AZ
6d ago
+2406-10

Aziz asked for a cmux NEXT stable pipeline beside the nightly: it runs once a night with stricter checks before it ships, and it uses no M6 Mac mini. The interview settled these choices:

  • App: a separate app, "cmux NEXT Stable".
  • Candidate: the newest nightly build, with no soak-time floor.
  • Gates: full Swift suites, perf budgets, crash-class rules, launch smokes of the signed DMG, and a soak and crash check.
  • Hosts: most work runs on macOS 27, with a sanity check on macOS 26.
  • Infrastructure reused: the existing bucket, Sparkle key and environment.
  • Schedule: publishes overnight, starting at 9 pm Pacific.
  • On failure: a rolling issue, plus a manual rollback workflow.

What changes

stable-next.yml is new. A tick on main runs every hour from 21:00 to 06:59 America/Los_Angeles and dispatches one publish run on nightly-next. It stops once a stable build has shipped that night, or after 3 failed runs. Each run:

  1. plan: takes the newest item on the cmux NEXT feed that Nightly apps are offered. The build number <run id><attempt> points to its nightly.yml run, whose head is the commit every gate tests. It refuses a pulled build or one with a fail verdict.
  2. gates (in parallel):
    • crash-class: the cmux-next-crash-class.yml rules on the whole tree, plus the trap ratchet.
    • swift-suites: every CmuxNext test target in 4 shards, run as build-fleet steps (package-test-lane.sh suite). The steps carry the labels macos-27 and ram48. Every M4 Pro worker has ram48 and no M6 worker does, so the M6 minis are excluded. Each shard retries once.
    • perf-budgets: split-spring.sh --suite=budgets --gate on an exclusive macOS 27 M4 Pro measure worker, compared against the apple-m4-pro-macos27 baseline.
    • sign: on Blacksmith macOS 26, the nightly's DMG is re-signed as cmux NEXT Stable, then notarized and stapled, and the Sparkle item is generated.
    • smoke-27 and smoke-26: the fresh-home launch smoke of the signed stable DMG. One runs on an owned M4 Pro mini on macOS 27, the other on Blacksmith macOS 26.
    • soak: the build is still on the feed and was never pulled, and Sentry shows no new unhandled issue first seen in cmux-next@<version>+<build>.
  3. publish: only the DMG whose sha256 both smokes tested goes up. The DMG uploads write-once to files-next.cmux.com/nightly-next/stable/. The feed (new item plus the 2 newest earlier builds, all tagged cmux-next-stable) passes nightly-next-guard.py appcast and is written by compare-and-swap.
  4. report: a failed run opens the rolling cmux NEXT Stable blocked issue, or comments on it if it is open. The next publish closes it.

stable-next-rollback.yml (inputs reason, bad_build/to_build, dry_run) pulls a bad stable build, so the feed serves the previous one. A rollback can never empty the feed.

Identity (scripts/ci/stable-next-identity.sh) re-signs the nightly's exact bits:

  • Name and bundle id: cmux NEXT Stable, com.cmuxterm.app.nightly.cmux-next-stable.
  • Entitlements: cmux.next-dev.entitlements, so no provisioning profile is needed.
  • Icon: the regular cmux icon, as AppIcon.icns.
  • Sign-in scheme: cmux-next-stable.
  • Feed: SUFeedURL is .../nightly-next/stable/appcast-arm64.xml.

Two of these differ from the interview's draft:

  • Bundle id: BuildIdentity treats any com.cmuxterm.app.<x> id as a DEV build. com.cmuxterm.app.next.stable would therefore get DEV behaviour and the dev Sentry environment. The nightly prefix keeps nightly behaviour and still gives the app roots of its own.
  • Feed path: the feed stays under /nightly-next/ because UpdateFeedResolver picks a feed's channel from that path segment. A /stable-next/ URL would be classified as classic stable, which rejects every tagged item. TipAppTrackTests.cmuxNextStableIsANightlyTrackOnItsOwnFeed pins this.

Allowlists the new identity needs:

  • CmuxAppCodeCheck.identifiers (the Computer Use host check) and its test.
  • verify-app-bundle-channel-metadata.sh next-stable and notarize-nightly-dmg.sh CMUX_CHANNEL=next-stable.
  • cmux-next-stable in web/app/lib/native-callback.ts.

Repo rule: "Never add CI gates… CI jobs or steps" (Lawrence 2026-10-23) is about slowing pushes and nightlies. This PR adds nothing to any push, pull request or nightly run: the new workflow runs only on its schedule, a dispatch, or a push to stable-next-dry-run. tests/test_stable_next.py is registered manual, following the test_nightly_next_launch_smoke.py precedent. Calling this out because it is a judgment call against the letter of that rule.

Docs: docs/ci/stable-next.md.

Before the first publish (not in this PR)

  1. A promotion carries this to nightly-next; release-next deploys only there.
  2. stable-next.yml and stable-next-rollback.yml must land on main, because the schedule and dispatches register only from the default branch. That needs a separate PR to main with the same two files.
  3. Deploy the web allowlist change. Until then, sign-in from cmux NEXT Stable falls back to the cmux scheme.

Observed / Not run

  • Ran: python3 tests/test_stable_next.py (15 pass). Ran stable_next.py pick against the live feed, which picked build 4099184203401 → nightly.yml run 40991842034 → 0b45ce8ec311. Ran actionlint 1.7.12 on both workflows (clean), check-bash32-parse.sh, prepush-gate.py (no new violations), and test_ci_cmux_ci_client_guarded, test_ci_macos_xcode_selection (exemptions added for the submit-only and smoke jobs), test_runner_label_policy, test_ci_workflow_guards_are_wired, test_ci_reusable_workflow_permissions, test_ci_blacksmith_linux_failover and test_ci_nightly_no_hosted_mac, all passing.
  • test_ci_fork_runner_routing fails on cmux-next.yml:1251 (vars.CI_SESSIO…), a file this PR does not touch. Base finding.
  • Dry run on stable-next-dry-run: link and result in a comment below.
  • Not run: a real publish. It needs release-next on nightly-next, which this PR cannot reach before it lands. Signing, notarization, the Sparkle item, the R2 upload, the Sentry read (the token's scope is unverified) and rollback are exercised only by the first real run.

No capture: CI pipeline only; the app's UI is unchanged.

Performance

No perf numbers: CI pipeline only, no app code. The perf-budgets gate this adds runs the existing scripts/cmux-next/perf-budgets.py --gate against its stored baseline.

Changelog

none

🤖 Generated with Claude Code