ci(nightly-next): a fresh-home launch smoke harness for the signed app

Merged
TE
10d ago
+868-0

Leo, on the NEXT nightly that shipped a launch crash: "how did this even happen in the first place is this prevented in the future?"

A NEXT nightly shipped a crash on first launch. Commit 4b5c86ed (the Home first-run panel's Chief intro) threw NSInternalInconsistencyException in HomeFirstRunView.applySpacing whenever the intro was not arranged, and 1daea603 fixed it. Every static nightly check passed because nothing launched the built app before the feed offered it. This PR adds the harness. The follow-up PR wires it into the nightly-next publish gate.

What it does

scripts/ci/fresh_home_launch_smoke.py (--dmg DMG | --app APP) [--seconds 30] [--out DIR]

  • Copies the app out of the DMG. The DMG is mounted read-only and always detached, forced if needed, even after a timed-out attach.
  • Launches the app the way a first install would:
    • an empty HOME, with CFFIXED_USER_HOME set to it, so Application Support, caches and logs start empty;
    • a private TMPDIR;
    • none of the runner's environment, so no tokens and no CMUX_* routing reach the app;
    • no defaults. cfprefsd keeps defaults per account whatever HOME says, so the bundle's domain is exported and deleted under a per-bundle lock, then imported back afterwards. The set-aside copy is kept beside the lock, so a run that is killed (a step timeout) leaves it for the next run to put back first. A failed delete or import means exit 3 (fail closed), and the copy stays.
  • Passes only when all of these hold:
    • the app stays up 30 s;
    • no crash report or run.exception is written;
    • Home renders in a visible window. This is read from the app's own signals: snapshot.get on the release control socket shows Home's page tab (home-dock) selected in a visible window's workspace, and the first_window_frame_committed launch mark arrives on CMUX_NEXT_LAUNCH_MARKS_FD. A screenshot is kept as evidence only and never decides the verdict.
  • Exit codes: 1 for a crash, an exit, or Home never rendering; 3 when the smoke cannot run (no Aqua session, or a signing keychain still present). A gate treats anything other than 0 as a failure.
  • Bounded: 30 s up and at most 90 s for Home to render.
  • Cleans up: stops the app, then every process of this user whose command line names its private root, by exact PID; deletes the root. --out keeps report.json, the app output, the .ips / run.exception, the app's system-log error lines (where AppKit reports an assertion) and the screenshot.

Proof, on a fleet mini (cmux NEXT nightly DMGs from the feed and release)

  • A clean build (4059411161803, commit cc0d757, before 4b5c86ed) passed. Home rendered in a visible window 5.1 s after launch, and the app was up 30 s with no crash.
  • A crashing build (4062224162001, commit 0c83363, contains 4b5c86ed and not 1daea603) failed:
    FAIL: the app exited (signal 11) 2.8s after launch and FAIL: crash report: cmux-2026-10-27-224207.ips.
    The system log has *** Assertion failure in <private>, NSStackView.m:873 at the moment of the crash, which is the NSStackView custom-spacing assertion 1daea603 fixed.
  • After each run, the account's defaults for the bundle matched the pre-run export and no process or temp root was left.

Tests

tests/test_fresh_home_launch_smoke.py (linux-guard lane, run by a new ci-guards release-notary step, with stub apps):

  • a stub that aborts at launch fails (signal 6);
  • a healthy stub passes. It serves snapshot.get with Home selected and streams the launch marks;
  • a stub that stays up without Home fails;
  • the app sees an empty home and no runner secrets;
  • the defaults are set aside and put back byte for byte;
  • a detached helper is stopped by the run's root;
  • a signing keychain in the job means exit 3;
  • crash reports are matched by executable name;
  • defaults a killed run left aside are put back first, and a domain that did not exist is empty again afterwards;
  • a failed export, delete or put-back is exit 3. A failed export or delete never launches the app, and the set-aside copy is kept.

Size: about 870 lines, of which about 400 are this harness's own tests and stub apps. The rule says to keep a change's tests in the same pull request, and the harness can't be split further without leaving half a cleanup path. The gate wiring is already its own PR (#26723).

Agent session: agent session

šŸ¤– Generated with Claude Code