fix(next): Ctrl-`, Cmd-D and New Row always give a terminal; a refusal is never silent; no stand-in id reaches the daemon (P0)

Merged
LA
7d ago
+1505-90

Summary

P0 from dogfood v2.2.9-opt (Lawrence, a 4 h session): "at some points I can no longer make new terminals via Ctrl-` or Cmd-D".

What the person saw. A new terminal (Ctrl-`), a split (Cmd-D) or a new row appeared for a moment and vanished, or the keys typed for it ran in the old terminal.

Cause, three faults in one family: the app took a new terminal back while the daemon still made it, or never asked for it correctly.

  1. A client clock decided whether the terminal exists. Ctrl-shows a stand-in tab at once and sendsnew-tabwith a client-chosen terminal id. That request had the 5 s spawn deadline. When the reply came later (it queued behind a slow request on the app's connection),DaemonService.intendNowReporting` took the stand-in back while cmux-tui still made the tab. New Row had the same 5 s deadline. Splits, columns and a new workspace's first terminal gave up after fixed 120 s or 10 s budgets.
  2. A stand-in id was sent to the daemon. A second Cmd-D (or a Ctrl-`) on a split still in flight targets the first split's stand-in pane. The app sent that stand-in's handle, which the daemon does not know ("unknown pane"), so the second pane vanished. The bench-arms run 02dcba2e saw this 12 of 12 times.
  3. The refusal was silent. A refused optimistic create only logged; the person saw the pane vanish with no reason (refusals []).

Fix.

  • A new-tab and a new-row (cmux-tui replays neither when sent again) are sent once and wait for their reply or the connection's end: DaemonConnection.requestAwaitingStart (CmuxNextDaemon/Connection/DaemonConnection.swift:154), used by newTab (DaemonConnection+Commands.swift:136) and RowCommands.newRow (RowCommands.swift:28).
  • Keyed creates the daemon replays (split, new-pane, new-pane-right, create-workspace, create-terminal) wait through late replies with no give-up budget. A re-send only repairs a lost request, so it is spaced by full-jitter exponential backoff from 1 s, capped at 30 s (R-L4): waitingThroughLateReplies and lateReplyBackoff (DaemonConnection+LateReplies.swift:101, :116, Backoff.fullJitter). Every new workspace with a terminal (Cmd-N, Open Folder, agent fork, agent teams, incognito) makes its first terminal through one path, WorkspaceCreation.firstTerminal (WorkspaceCreation.swift:78).
  • Rules (a) and (b) below.

Checklist (cmux-ad rules for this P0)

  • (a) A refusal is never silent. A refused optimistic create (Cmd-D split, Ctrl-` New Terminal Tab, New Row, New Column) rolls its stand-in back AND shows the refusal in the refusal HUD with a log line:
    • ActionRegistry.trackRollingBack (CmuxNextActions/ActionRegistry+Effects.swift:80), used at Handlers/PaneHandlers.swift:174, PaneController+Intents.swift:202 and WorkspaceContentController+Intents.swift:173.
    • The text is localized at the app boundary: ActionWorkFailure.create (Handlers/MiscHandlerStrings.swift:120), key handlers.misc.failed.pendingPaneNotCreated in 21 locales.
    • A create the person closed first (PaneSplitCancelled) is not a failure.
    • Test RefusedCreateIsNeverSilentTests.
  • (b) A stand-in id never leaves for the daemon.
    • Creates wait and resolve: every create that targets a pane resolves it at send time with DaemonStore.daemonTarget (CmuxNextDaemon/Store/DaemonStore+Intents.swift), used by PaneSplitCommand, NewColumnCommand, NewRowCommand and NewTerminalTabCommand. It returns the daemon's pane that took the stand-in over, or throws PendingPaneNotCreated. These creates run in the workspace's create chain (pendingCreate), so each goes out after the create that makes its target; the wait is the earlier request's own outcome, no timer. A Ctrl- tab on a stand-in follows it to the daemon's pane (IntentOverlay`).
    • daemonTarget also throws PaneNotReady when the pane it finds is still a stand-in (the pending pane of a Cmd-N workspace has no resource id yet).
    • One boundary for every sender: DaemonRequestExecutor.perform scans each encoded line before it is written (one byte search for a decimal number >= 2^62, a JSON parse only on a hit, keys pane/surface/target/anchor) and throws PaneNotReady for any request that names a stand-in, creates included, in every build. There is no daemon round trip and no DEBUG stop (the earlier TargetResolvedCreate exemption is removed: it crashed DEBUG on Split Left of a pending split and sent the stand-in in release). Cmd-N's workspace stand-ins (2^62 + 2^61) are caught too.
    • Shown once, for every sender: the refusal is reported in DaemonConnection.request (reported), the one place every request goes through: the DaemonService funnels and the handlers that call the connection themselves (a typed New Tab page command or a cmux.json newTabInCurrentPane, New Tab in Group, the non-optimistic split/column/row, agent fork). With no open action scope (DaemonCommandScope.isOpen), the connection calls its observer, which DaemonService sets on each connection it makes (observePaneNotReady): the notice shows on the next frame through the drain batcher (log line + HUD "This pane is still opening; try again in a moment.", handlers.misc.failed.paneNotReady, 21 locales), and the error is marked shown (PaneNotReady.shown, carried as ActionWorkFailure.shown). The registry (ActionRegistry+Effects), the handler reporters (AppActionContext.report, StashHandlers.report, agent create) and agent fork skip a shown failure, so it never shows twice. An open scope (CLI, MCP) answers the typed unavailable refusal in the app's language (DaemonService.workFailure) instead. The optimistic creates pass PaneNotReady up, not the generic "the daemon refused the split". Clear history does not report it to the crash reporter.
    • Tests SplitOfPendingSplitTests, NewTerminalOnPendingSplitTests, StandInReferenceTests (with StandInLineTests), PaneVerbOnPendingPaneTests (zoom shows once; Split Left on a pending split shows once, sends nothing, makes no pane; request and perform each show once).
    • Known limit (open decision for cmux-ad, STATUS 611/612): a create on the pane of a Cmd-N workspace that is still pending (Cmd-D, Cmd-Shift-D, Ctrl-`, New Column, New Row, Split Left/Up, a command typed on a New Tab page) is refused with "still opening" and is not made; the person presses it again once the workspace shows. Before this PR it crashed DEBUG or sent the stand-in ("unknown pane", no notice). Follow-up: one seam where every pane verb waits on the create chain for its pane.
    • Known open race, landing anyway by cmux-ad decision (2026-10-30 18:0xZ): issue https://github.com/manaflow-ai/cmux/issues/28436, P0, owner the p0-port agent (a2c6445561ebeab90), who owns priority (a), the basic tasks. After a quick double Cmd-D, the second pane exists and has focus, but in a rare case the line typed into it does not run there.
      • Measured rates. Base (no fix): the second Cmd-D made no pane 12 of 12 (bench-arms job 02dcba2e1c9e447c7cad0594, https://github.com/manaflow-ai/cmux/pull/28221#issuecomment-6520438640; red proofs 89d99910 and bc080642 above). With the fix: 2 failures in 13 runs, about 1 in 6 to 1 in 7. At head 69167de6264: 05563df226b8bea654bde31f 1 fail, 216230c32be6cf456369136d 5 pass, 17418039cdae019ea11d141f 1 pass. At 3776515e8ae: 99c595e46b863d756e5dd354 1 fail and 4 pass, green 79fe4d7f 1 pass. At 642e8790b63: green f2d57d1a 1 pass. At the merged head 47a86e155f0 (feat-cmux-next merged in, 2026-10-31): 25b930845f86c6948f4e3ffc 1 fail, 82b745f89e19448e86092443 3 pass (1 in 4). Re-run at the pre-merge head 69167de6264 the same hour: 54119ff5351b870500661fc1 2 fail, 1 pass. Every failure is the same race (SplitOfPendingSplitTests.swift:14:6, wait-for timeout after both splits confirmed), so the merge made it no worse. All feat runs with the fix: 5 failures in 20.
      • Still silent: in both failed runs the test recorded only the wait timeout. Both splits confirmed, no refusal reached the refusal observer, the second pane had focus and the window's first responder was in it; the typed line did not show and nothing said why. The next commit after this PR, in #28436's PR, makes it visible or removes it (the keys reach the new pane, or one notice says why).
      • Next step today (in #28436): the diagnostic red that prints, at the failing moment, every pane's screen, the creation hold's end (focusLanded or limitReached), whether the surface was attached when each key was sent, and the HUD count.
      • Known-failure entry: SplitOfPendingSplitTests.aSecondSplitOnAPendingSplitsPaneMakesASecondPane, owner the p0-port agent (a2c6445561ebeab90), #28436, expiry 2026-11-01. The suite stays enabled (no skip, #if or withKnownIssue). CmuxNext Swift suites have no per-suite known-failure registry (scripts/cmux-next/quarantine.json holds fast-tier entries and car-guard entries per test target), so the entry is here and in #28436.
    • Known limit (follow-up, agreed with the lead): in a CLI or MCP run, New Tab in Group (TabGroupHandlers.swift:114-125) and the non-optimistic split / new column / new row (WorkspaceContentController+Intents.swift spawnPane, :178-190) call the connection in a plain task with no ticket and only log a failure, so the run answers "ok" when the refusal comes, and the app shows no notice while that run is open (the connection leaves an open scope's refusal to the run). This was true before this PR for every error on these paths. Follow-up: https://github.com/manaflow-ai/cmux/issues/28394 (send them through a DaemonService funnel and track the work; a red CLI test for each of the 4 actions first).
  • R3 invariant, checked in DEBUG: a terminal stand-in is never taken back on a client deadline (DaemonService+StandInInvariant.swift:33), and no late-reply wait ever ends on a late start (checkNoLateStart).
  • Creation hold ends on a signal (R-L5). The hold records why it ended (CreationInputCoordinator.HoldEnd: focusLanded, limitReached, focusLeft, windowClosed, creationFailed). A limitReached end is logged (:211). NewTerminalOnPendingSplitTests asserts the Ctrl-hold ended withfocusLanded`.

Testing

All reds failed on their own assertion (not on build or runner), except c0fe852f, which failed by the base crash (the bug) as its row says. The zoom row is a guard test, not a red.

SuiteRed refRed jobFailure
NewTabLateStartTests7340d853d95 (base + test)eefaa476timed out: new-tab (no reply within 0.2 seconds)
NewRowLateStartTestsproof/p0-newterm-row-red c83d58407c555feecedtimed out: new-row (no reply within 0.3 seconds)
WorkspaceWithTerminalLateCreateTestsceac05ed3f2 / proof/p0-newterm-firstterm-red2 0230e76236d6a9f1764 / 79167017old 10 s budget gave up / create-terminal ... 0.5 seconds
SplitOfPendingSplitTestsproof/p0-newterm-twosplit-base (origin/feat-cmux-next + test) 9081d397e6289d99910new confirmed terminal panes 1 of 2; refusals []
SplitOfPendingSplitTestsproof/p0-newterm-twosplit-red2 (head minus the two-split fix) c0cd1a7fff3bc080642same
RefusedCreateIsNeverSilentTestsproof/p0-newterm-rule-a2 f00df28d58e5ed751f9the refused split showed nothing (HUD shown 0, was 0)
StandInReferenceTests, NewTerminalOnPendingSplitTestsproof/p0-newterm-rule-b2 938bd6e153794d09943stand-in not found (lines 9-11); the Ctrl-` tab never reached the split's pane
PaneVerbOnPendingPaneTests442f88cbcca (test before the fix)95d2824czoom on the pending pane showed nothing (HUD 0, was 0)
NewTerminalOnPendingSplitTests (with the hold-end check), StandInReferenceTestsproof/p0-newterm-rule-b3 921450584d9e74936a8the Ctrl-` tab never reached the split's pane (line 33); stand-in not found (lines 9-11)
PaneVerbOnPendingPaneTests, Split Left (review #6 B1)5e9beea227f (test before the fix)c0fe852fthe base crash, not an assertion: splitLeftOnAPendingSplitsPaneShowsOneNoticeAndSendsNothing hit DaemonRequestExecutor.swift:68: Fatal error: split names stand-in pane=... (the DEBUG stop review #6 B1 named; the crash is the bug: Split Left sent a stand-in). It killed the process, so the funnel test did not run in this job.
PaneVerbOnPendingPaneTests, every funnel (review #6 B2)proof/p0-newterm-r6-b2-red d8cb74d6902 (head minus its rule (a) part: DaemonService.ended, ActionWorkFailure.shown, the reporters, the create rethrow; the B1 boundary kept; test files identical to the head's)3754ffcceveryFunnelShowsAPaneNotReadyRefusalOnce failed on its own assertion (PaneVerbOnPendingPaneTests.swift:81:44, the until condition: request on a pane still opening showed no notice), no crash. Zoom and Split Left passed there (zoom goes through DaemonService.send, Split Left through perform and trackRollingBack; both showed the refusal before this fix).
PaneVerbOnPendingPaneTests, typed new tab on a pending pane (review #7 B1)33e56b395b8 (test before the fix)3cac6787aCommandTypedIntoANewTabOnAPendingPaneShowsOneNotice failed on its own assertion (PaneVerbOnPendingPaneTests.swift:110:44, the until condition: the typed new tab on the pending pane showed no notice), no crash; the other 3 tests of the suite passed
PaneVerbOnPendingPaneTests, zoomnone for this delta(95d2824c at 442f88cbcca, the earlier red)a guard test, not a red: it passes at 5e9beea227f and at d8cb74d6902, so it does not prove this fix. Its new "exactly once" line guards against a second notice from ended plus the registry.

Green at the earlier head 670131ca0ae: cba1a2416c5fb6b6c721bf25, 11 suites, 28 tests, none skipped. Green at 321277db665: c311521d (12 suites). Green at 3776515e8ae: 79fe4d7f (12 suites). Green at 642e8790b63: f2d57d1a (12 suites, all ran, 22 tests by the run summaries, none skipped). Green at this head 69167de6264: 05563df226b8bea654bde31f (12 suites; result added when it ends). Suites: NewTabLateStartTests, NewRowLateStartTests, SplitLateStartTests, WorkspaceWithTerminalLateCreateTests, TerminalStandInRollbackInvariantTests, LateReplyBackoffTests, SplitOfPendingSplitTests, RefusedCreateIsNeverSilentTests, NewTerminalOnPendingSplitTests, StandInReferenceTests, CreateFailureTextTests, PaneVerbOnPendingPaneTests. Compile (CmuxNext + app, test targets) at this head 69167de6264: b3fd053c6fe4868e52de9d99 passed (e35af5f2 at aa8769f550f failed on one Sendable error, fixed in 69167de6264).

The proof refs are test-only proof/* branches listed in stopped.txt as never merge; each has test files identical to the head's.

Observed / Not run

Observed:

  • The bug on the dogfood build 7c900e1fa082 (ref 8a038629cd5d) in probe journeys on exclusive capture leases (cmux17, cmux20, cmux21): after Cmd-T, Ctrl-, the typed line ran in the old terminal or no terminal showed, 5 times. App log each time: new-conversation-tab ... no reply within 2.0 seconds, then new-tab rejected: timed out ... 5.0 seconds, then new-tab failed: the daemon refused the new tab`.
  • The holder of that stall, from samples during it (cmux20): the daemon's request workers were idle; one of the app's two daemon readers was parked in TerminalEventQueue.parkReader() for the whole 3 s, so replies waited unread behind one terminal's output. Filed for pa-b1 as TRANSPORT MULTIPLEX (handoff/followups.md); not fixed here.
  • All red and green jobs above.

Not run / only read:

  • The new journey newterm-after-new-tab-page (Ctrl-, Cmd-D and terminal-first Cmd-N right after Cmd-T) is on the perf port cp/fix-p0-new-terminal-always`, where the journeys live; it runs there on an optimized build.
  • The replay of held keys through NSApp.sendEvent (keys typed while a creation hold waits) is not covered in a non-key test window; the test types after the hold and asserts the hold ended with focusLanded.
  • SSH and Cloud remote links have no heartbeat, so a waiting create there is bounded only by link liveness; filed for pa-b1 (REMOTE LINK LIVENESS), with "a reply wait must honor the caller's cancellation".
  • Real dictation and a 2 h session were not run (the targeted repro was enough). The diagnostics' spare in "loading" and the PageWebView first responder were read in code and ruled out.

Follow-ups (separate PRs, mine): one seam where every pane verb (zoom, rename, swap, a New Tab page, the creates on a pending Cmd-N pane above) waits on the create chain for its pane instead of being refused, and where a create waits only when its target is a stand-in (review #4 N2, N3); a per-pane error state with its own strings and capture; a visible "starting…" label on a pending stand-in; the scripted newTab that made no sidebar row (followups.md, STATUS 578).

Performance

No perf numbers: this PR changes when a new terminal is made or refused (no client deadline, a stand-in id never sent), not how fast anything runs; "perf" here names the perf/chrome-parity branch, where the port runs the journey newterm-after-new-tab-page on an optimized build. The request boundary adds one byte search per request line (a JSON parse only on a hit), with no allocation on the common path.

Capture

No capture: the change is when a new terminal is made or refused under a slow daemon reply or a split still in flight, a race of a few milliseconds that no capture scene or tour holds; a still would show a normal terminal. The only new pixels are the existing refusal toast with the new localized text, which the tests assert by string key (MiscHandlerStrings.paneNotReady). The journey newterm-after-new-tab-page on the perf port drives the real keys on an optimized build.

Changelog

Fixed: Ctrl-`, Cmd-D and New Row always give a new terminal, also when the daemon answers slowly or a split is still being made; a refused one now says why.