PR #22919 - manaflow-ai/cmux

Summary

Incident 2026-10-20 (nightly 3951982493001): Lawrence clicked Restart to Update. The app looked frozen. Process 5342 exited 28 s after the click; launch looked slow as well. Bead cx-p00l3.

Root cause (unified log, log show of pid 5342 and the Sparkle tools):

Time	Event
19:28:43.545	Click. Every window goes transparent (`UpdateWindowHandoff.hide` at `willInstallStaged`).
19:28:50.736	Windows come back: Sparkle aborted the install after its 7 s first-installer-message timeout (`SPUInstallerDriver` `FIRST_INSTALLER_MESSAGE_TIMEOUT`, "An error occurred while starting the installer"). The installer `Autoupdate[6160]` had staged the update at 19:00:14 and logged nothing in this window.
19:29:02.48	A click opens a URL in the default browser (Download Manually on the failure sheet).
19:29:09.8	A click (Retry).
19:29:10.971	`Handling Quit AppleEvent` (Sparkle's agent).
19:29:11.257	`CEF shutdown complete`, `replyToApplicationShouldTerminate:YES`. The quit itself took 0.29 s.
19:29:11.46	New app spawned; window on screen at 19:29:13.6.
19:29:13 to 19:29:33	New cmux-tui daemon (version handoff) adopts 113 terminal hosts one by one before it answers; terminals blank until then.

So the 28 s was not the quit: it was 7 s of no visible windows, then the failure sheet and a second click. The launch delay is the daemon's serial host adoption (follow-up cx-p00l3.1).

Changes

Restart to Update: the windows hide at updaterWillRelaunchApplication (Sparkle calls it only on the path that sends its installer to stage 2 and quits the app), not at the click, and only for a relaunch the person asked for (hidesForRelaunch, called when no automatic install is in flight). A click whose install fails keeps the windows on screen with the Installing state, then the failure. An install nobody clicked (Sparkle's automatic driver, the relaunch behind the person's app) leaves the windows as they are, as before; a declined relaunch behind and an automatic failure still call installAbandoned. The willInstallStaged hook, which nothing set any more, is gone.
Quit hard bound (keep quits): a quit that keeps sessions ends 4 s after it is decided whatever any step does (QuitDeadline.keepQuitLimit). An End quit has no deadline until its engines begin to stop (below). Each save step has a limit (QuitSaveSteps in CmuxNextControl, run by a QuitStepLimiter instance that QuitCoordinator owns): browser records 300 ms, windows 500 ms, sidebar 300 ms, Home's cache 400 ms, agent chat composers 400 ms, then terminal frames, agent images and remote snapshots 200 ms each. The person's own data saves first. Cloud's quit work (Codex broker stop, VM credential revokes) starts first and runs beside the saves, so a slow network never delays a save. If every step hangs they take 2.6 s (worstCase; Cloud's in-step wait is 100 ms, the late-step wait gives it the rest); with the 500 ms quit mark (QuitDeadline.quitMarkLimit), the 300 ms "Don't ask again" write and the 200 ms reply margin, 400 ms remain.
Late steps still finish: a step past its limit keeps running beside the next steps, and the quit waits for every such step before AppKit's reply, for what is left of its deadline less the 200 ms reply margin (QuitStepLimiter.finishOverruns, QuitDeadline.timeLeft; async, never a main-thread wait). An End quit also waits up to 2 s for a late save (not Cloud's step) before its sessions and daemon stop. In a keep quit, late saves get what the deadline leaves less the engines' 2 s close wait and the reply margin, so Chromium still closes its pages and flushes cookies and storage; a save still late then runs on beside the engine stop, and the wait before AppKit's reply covers it. A closed page has no session to save (BrowserRestoredSession), so a late browser-records save writes nothing over the good record the daemon holds (written after each navigation). In an End quit the deadline is armed only as the engines stop: a save still late after its 2 s End wait runs on beside the engine stop (its daemon has stopped). Chat pages are asked for their composer state within 1 s (AgentPaneRelaunchWake.askLimit, as on feat): the page first waits up to 500 ms for its typed draft to reach acpmux (draftSettleMs in schemas/agent-pane/wake-state.json, which composerDraft.ts and the host's test both read), and the quit waits for the composer step past its 400 ms limit. A page that does not answer in time is logged (agent-pane.relaunch-wake). Engines' close wait 2 s (was 3 s); an End quit ends 3 s after its engines begin to stop (was 15 s). The window records flush before the incognito close, and a windows step past its limit stops saving at once. Each step writes a quit.* launch mark.
Diagnostics: the quit origin and choice log at notice level, so they stay on disk. The updater log (where the incident's Sparkle error text was, lost because it was info level) moves to notice level in the separate high-tier PR #22920.
Bench script (not CI): scripts/cmux-next/quit-launch-bound.py (behavior check: real tagged app, control socket, launch-mark pipe, loopback pages with beforeunload handlers, sample of every quit and launch). It fails on a quit over 6 s, a launch over 3 s to the first window or 8 s to the first live terminal frame (from main()), a launch main-thread stall over 2 s, or a restored session with fewer workspaces or tabs than before the quit. This PR adds no CI step or job (repo rule: no new CI gates); run it yourself on a GUI host.

Numbers (mac26, 8 workspaces / 32 terminals, 5 browser tabs, 1 agent pane)

Quit (request to process exit)	base 7819daf2	this PR
interactive (Cmd-Q / quit Apple event), live pages	1.91 s	1.71 s
scripted (`cmux app quit`)	0.13 s	0.12 s
SIGTERM	0.12 s	0.11 s
engine stop wedged on the main thread (`debug.quit wedge_engine_stop`)	15.04 s	4.04 s (4 s deadline)
Restart to Update, click to old app exit (update-harness, real Sparkle)	85 ms	63 ms
Restart to Update, click to new window	1114 ms	1178 ms
Restart to Update, windows hidden	at the click (+24 ms, before Sparkle has the install)	at `will_relaunch` (+22 ms)

Launch (from `main()`, which comes about 1.5 s after process start)	base	this PR
first window frame	~0.45 s	0.43-0.49 s
first live terminal frame	~0.75-1.25 s	0.66-0.96 s
longest main-thread stall	0 ms	0 ms
after a daemon version handoff (28 workspaces, 112 terminals)	terminal at 14.7 s from process start; `daemon.ensure` 8.4 s	unchanged: follow-up cx-p00l3.1

Samples: sample every 2 s during each quit and launch are in the job artifacts (nx-remote mac26 jobs 1020-200458-a88a5f base, 1020-204732-4fdd44 this PR). Base wedged quit main thread, every sample: semaphore_wait_trap <- QuitDeadline.wedgeMainThread <- closure #6 in QuitCoordinator.complete until the old 15 s deadline.

Keep-quit time budget (4 s from the decision)

Quit mark ≤ 0.5 s (QuitDeadline.quitMarkLimit), "Don't ask again" ≤ 0.3 s, then the saves in order: browser records 0.3, windows 0.5, sidebar 0.3, Home's cache 0.4, chat composers 0.4, terminal frames 0.2, agent images 0.2, remote snapshots 0.2, Cloud 0.1 (2.6 s if every step hangs).
Late saves before the engines: until 4.0 − 0.2 (reply margin) − 2.0 (engineCloseWait) = 1.8 s after the decision. Then the engine stop, then the wait for every step still running until 3.8 s, then AppKit's reply.
Chat composers: the step starts after the quit mark and the four saves before it (in practice tens of ms; 2.3 s if every one hangs). Each page's ask is ≤ 1.0 s (the page's 500 ms draft settle plus 500 ms for a large chip) and the write is ≤ about 0.3 s, so it ends inside the 1.8 s wait when it starts within 0.5 s. A later one runs on beside the engines' close (the asks are async on the main actor, the write is detached) and the last wait covers it until 3.8 s (2.3 + 1.0 + 0.3 = 3.6 s, theComposerAskFitsTheKeepQuitDeadline). Only CefShutdown itself, which blocks the main thread, can delay its last page answers.
The trade-off when a save hangs: Chromium's close and its last cookie and storage flush keep their 2 s; a late save gets the time before and after it.

Cloud credential cleanup at quit (trade-off)

On feat, a quit awaited cloud.prepareForTermination() to its end before any save: the Codex broker's stop (SIGTERM, up to 2 s grace) and one revoke per VM it delivered to (each bounded at 3 s, about four or five app-server round trips: isLive, read access.json, read the mode file, remove, restore or remove). With no keep deadline, a slow network delayed the whole quit by that much.

Now Cloud's cleanup starts first and gets all the time the quit has left: the save steps' time, the engine stop, then what is left of the 4 s keep deadline (End quit: the 3 s engine-stop deadline) less 200 ms. On a normal network that is enough for the broker stop and a few revokes. When it is not (a slow network, several VMs, a broker that ignores SIGTERM), the deadline ends the process first, and access.json and the own mode file stay on the VM until the delivered token expires. That is the trade-off of a quit that always ends: the token's expiry is the backstop. plans/cmux-next/OWNERS.md has no row for the Cloud broker (CmuxNextCloud/Broker/); its author (#26170, #26212) is asked for an OK in a PR comment.

Risk tier

risk:high (updater). No default-off setting: both changes fix user-visible regressions (the click-time hide that left cmux invisible for 7 s in the incident, and a wedged quit that took 15 s), which docs/ci/risk-tiers.md ("Ship risky behavior dark") exempts. Automatic installs keep feat's behavior (no hide).

Review of c0756ab (FIX) and its fixes

Finding 1 (keep quit could end before the saves): fixed as above. Red 740d5c33d96, fix 0088e20ec76.
Finding 2 (an automatic install could stay hidden): fixed as above. Red bbfcfffe854, fix 0088e20ec76.
Stale text: this description, the title, restore-journey.py (4 s), the UpdaterService doc of the removed hook.
Fleet job ids are in a PR comment.

Review of 0088e20 (FIX) and its fixes

Finding 1 (a step past its limit was cut by the normal exit): the quit waits for late steps up to its deadline; End quits wait for late saves before the daemon stops; composer pages are asked within 250 ms. Red c2e92cb8e49, fix 19e931e1183, f95bdd0a0ef (the "Don't ask again" limit moves into QuitSteps), ce9cd46ad51 (Cloud's in-step wait 100 ms, so the limits fit with the reply margin).
Finding 2 (Cloud cleanup): the same wait gives Cloud the time left; the trade-off is above.
Finding 3: the budget test reads QuitDeadline.quitMarkLimit (the bound quitBegan passes); a test covers installAbandoned on the automatic failure branch.

Review of ce9cd46 (FIX) and its fixes

Finding 1 (the 250 ms ask was shorter than the page's 500 ms draft settle): the ask is 1 s again; the page's settle time is one shared value in schemas/agent-pane/wake-state.json; a page that does not answer is logged. Tests: theQuitAsksLongerThanThePageSettlesItsDraft (reads the JSON), theComposerAskFitsTheKeepQuitDeadline.
Finding 2 (a fixed 20 ms sleep and polling loops in the quit tests): the tests wait on the limiter's .limit mark and the manual clock's sleeper.
Note 3 (a late browser-records save during the engine stop): checked; a closed CEF page returns no navigation list, so its saved session would keep only the shown page. Every late save now lands before the engines stop.
Red 49563554e39, fix 654ab7183ff, merge of origin/feat-cmux-next 2cf3babb845 (the branch was 187 commits behind, and the web lint judged feat's own newer files; after the merge it passes).

Review of 6f77b0a (FIX) and its fixes

Finding 1 (this PR's own red "Web bundle key inputs"): schemas/agent-pane/* added to BUNDLE_INPUTS in scripts/ci/cmux_next_route.py and schemas/agent-pane/** to on.push.paths of cmux-next-web-bundle-publish.yml. These are two path lines in two existing filters: no new CI gate, job, step or test. bash tests/test_web_bundle_key_inputs.sh: web-bundle-key inputs: ok (it failed before); python3 -m unittest tests.test_cmux_next_route: 111 tests OK.
Note 2 (a hung save took the engines' close time): late saves before the engines end at deadline − reply margin − engineCloseWait; a closed page has no session to save. Tests: aHungSaveLeavesTheEnginesTheirCloseWait, aClosedPageHasNoSessionToSave.
Nits: aHungCloudStepHoldsNoSave waits on a signal; the QuitSteps doc and this description now say what an End quit does.
Routing d49e188e628, red 243c9d3333f, fix c99b21a4ff9.

beforeunload

App quit does not show a page's question, but Chromium's forced close (cmux_shim_close_all, CloseBrowser(true)) still runs each page's beforeunload handler: the test pages' beacons arrive about 0.5 s after close-all. The quit waits for it at most the 2 s close wait. Follow-up cx-p00l3.2 (renderer fast shutdown in the shim).

Changelog

Fixed: Restart to Update never leaves cmux invisible while the installer starts, and a quit that keeps your sessions always ends within 4 seconds.

Observed / Not run

Ran (at an earlier head) scripts/cmux-next/quit-launch-bound.py on mac26 (GUI host) with fleet-built apps: base 7819daf2 + check (red: wedged quit 15.04 s), this PR (green, 2 rounds of interactive, scripted, signal, wedged; session census kept: 9 workspaces, 32 terminals, 5 browser tabs, 1 conversation, 1 page).
Ran --handoff --workspaces 28 on base and this PR: shows the daemon adoption delay (not changed here).
Ran the update harness (scripts/cmux-next/update-harness/job.sh, real Sparkle update between two signed copies) on base and this PR on cmux-lawrence-2: both relaunched.
Gate (gate-run.sh): CmuxNext suites QuitSessionsTests, MainThreadWatchdogTests, MainThreadWatchdogPolicyTests, SilentBackgroundUpdateTests, UpdateCheckFeedbackTests, UpdateReadyCardLayoutTests, RestorePreviousLaunchTests, CrashNoticeTests on Xcode 26.6 (cmux-ci), the Release compile, check-concurrency, godfile check.
Not run: the Sparkle failure path itself (a second session asking a live installer to extract again, Sparkle's 7 s timeout). I found it from the unified log and Sparkle 2.9.5 source; I did not reproduce it (follow-up cx-p00l3.3). The fix is the hide timing, which the update harness covers on the success path only.
Not run: Xcode 27 compile of the app target (the fleet dev build compiled it on the Linux pool).

🤖 Generated with Claude Code