ci(nightly-next): a post-publish launch smoke that rolls a crashing build back (not a publish gate)

Merged
TE
9d ago
+899-5

This is not a smoke gate. Publishing never waits on it.

Leo, 2026-10-23: "we want no smoke tests. No smoke tests, okay? Or at least, like, you know, previously we had the daemon with a smoke test literally block the entire nightly release, right? Like, that's unacceptable."

Leo, on the NEXT nightly that shipped a launch crash: "how did this even happen in the first place is this prevented in the future?"

A NEXT nightly (0c83363, build 4062224162001) shipped a crash on first launch, and nothing launched it once it was on the feed. With this PR, a launch crash like that never stays on the feed for more than one smoke run. The feed rolls back to the earlier Nightly build it keeps (#26710), and the crash is filed with its evidence.

Design

  • The feed publishes exactly as it does now. This PR doesn't touch nightly.yml: no publish job needs the smoke or reads its result.
  • After publication, .github/workflows/nightly-next-launch-smoke.yml runs as its own workflow. A failure in it can't fail or delay a publish run.
    • Every 10 minutes it reads the feed's newest build and smokes it once. The verdict is cached per build.
    • It launches the app from the feed's DMG with scripts/ci/fresh_home_launch_smoke.py (#26722): an empty home and no defaults, up 30 s with Home rendered. It runs in the console user's GUI session.
    • Routing: an owned Mac's side label first, Blacksmith macOS last. It is registered in the owned-pool rescue.
    • The job has no secrets and no environment.
  • Outcomes (scripts/ci/nightly_next_launch_smoke.py):
    • pass: nothing happens.
    • cannot-run: nothing happens, and it never rolls back. This covers no Mac, a failed download, no GUI session, a harness or infrastructure error before launch, and a check cut short.
    • crash: on both of two attempts, the app launched (the harness reports its pid) and then exited, wrote a crash report, or never rendered Home. Exit 1 without that launch evidence, such as a failed GUI-session hop or a harness setup error, counts as cannot-run. The crashed job dispatches nightly-next-rollback.yml with bad_build (the existing rollback, not a second copy of its logic). It then opens or comments on one nightly-next launch crash: build N issue with the harness failures, the app's system-log errors, the crash report's exception, and the app output's tail. The .ips file and the full log are in the run's launch-smoke-N artifact.
  • Idempotent; never past the kept build:
    • A rollback is requested only while the build is still the feed's newest and an earlier Nightly build is there to serve.
    • The rollback re-checks that on its own read of the feed (new nightly-next-rollback.yml input if_newest), so a build published after the crash was judged is never pulled.
    • A crash verdict is never cached, so the next tick asks again until the build is off the feed. A failed rollback is retried.
    • One crash job runs per build at a time, and the issue is deduplicated over every state.
    • A second crash report of the same build, after its rollback, does nothing.
    • It never pulls a newer build, and never a Dev-only one.
  • No new secrets, and no permissions beyond what nightly-next-rollback.yml already uses. The crashed job has actions: write (the dispatch), issues: write and contents: read, the same scopes the rollback's own fix-forward job has.
  • A schedule and a dispatch register only from the default branch, so manaflow-ai/cmux#26788 puts this workflow and its rescue registration on main. It lands right after this PR. Runs take their tooling from feat-cmux-next; a pull request's run takes it from its own head, smokes the newest feed build, and never rolls back or files.

Tests

tests/test_nightly_next_launch_smoke.py (linux-guard lane, run by a new ci-guards release-notary step):

  • Verdicts: a pass on either attempt passes; a crash needs both attempts. No attempt, exit 2, exit 3, or a check cut short after one crash is cannot-run.
  • Decisions:
    • pass does nothing;
    • cannot-run never rolls back;
    • a crash of the newest build rolls back to the kept build;
    • a second crash report of the same build, run against the feed nightly_next_feed_rollback.py plan wrote, does not roll back twice;
    • nothing rolls back past the kept build or pulls a newer one;
    • only a Nightly-offered build is served instead.
  • Evidence: the issue body carries the failures, the NSStackView assertion line, the .ips exception and the log tail.
  • Nothing waits on the smoke:
    • publish-nightly's needs never include it;
    • nightly.yml doesn't name it or the harness;
    • no other workflow names it, except the owned-pool rescue that re-runs a refused job;
    • the rollback is a dispatch of nightly-next-rollback.yml, and nothing here writes the feed;
    • there are no secrets and no cancel-in-progress: true;
    • the Mac job routes owned first, Blacksmith last.

The harness has already been shown, on a fleet Mac, to pass on build 4059411161803 (Home rendered at 5.1 s) and to fail on 4062224162001: signal 11 at 2.8 s, with *** Assertion failure in <private>, NSStackView.m:873 in the system log.

Agent session: agent session

šŸ¤– Generated with Claude Code