Fix intermittent warm workspace switch latency

Merged
AU
2mo ago
+2589-376

Summary

  • Ship permanent, recording-gated switch diagnostics: ws.switch.selection-commit, ws.switch.portal-show, ws.switch.portal-hide, ws.switch.renderer-realization, ws.switch.first-frame, ws.switch.interactive, ws.switch.window-focus, and ws.switch.window-become-key.
  • Make mounted-workspace reconciliation the sole owner of switch handoff. Portal, frame, and focus callbacks measure presentation and interaction; they never retain the source workspace or block the selected destination.
  • Reconcile the current model selection when SwiftUI delivers a stale/coalesced value, so rapid A → B → C changes cannot remount B or strand C's transaction.
  • Protect a terminal renderer from selection commit until mount reconciliation retires the source. Protection is request-scoped, cancels on mismatch/rapid reselection, and self-expires through a weak owner before any reclamation pass.
  • Tie the switch-time frame observer and rendered-frame notification demand to an RAII owner, so deallocation is a complete cleanup path even without an explicit window callback.

Root cause

The 0.64.20 handoff path split lifecycle ownership between model selection and presentation/focus callbacks. It retained the old workspace behind hasLoadedTerminalSurface()—a native-pointer proxy rather than a presentation guarantee—or a 150 ms timeout. During rapid or delayed SwiftUI delivery, an earlier selection callback could also reconcile after the model had already advanced.

That made mount ownership depend on timing: the destination was selected, but the old and new view graphs and portals could remain in a handoff state until a callback or timeout happened. PR #7231 deduplicated repeated portal-hide work and PR #7236 stopped hibernation presentation from restoring unrelated portals, but neither removed this split ownership.

The reported 0.64.20 build predates renderer reclamation, so reclamation is not claimed as the original cause. Current main does reclaim renderers, however, so the switch transaction protects its destination through the authoritative mount transition.

The new invariant is:

  1. The model commits selection and starts one request-scoped diagnostic transaction.
  2. Any SwiftUI selection callback reconciles the current model value once; an intermediate delivered value never becomes a second source of truth.
  3. Mount reconciliation selects the destination and retires the source synchronously; it never waits for portal, frame, focus, or elapsed time.
  4. The destination renderer remains protected only through that mount transition: the unconditional selection callback calls sourceDidRetire immediately after reconciliation, even when the source workspace model was already deleted. A transaction mismatch cancels immediately, and a dead coordinator's weak owner makes any missed release ineligible for protection.
  5. Portal, first-frame, and interaction signals finish signpost intervals only. A missing signal cannot strand an old workspace or block another switch.

No sleep, polling loop, loaded-pointer proxy, callback-owned mount decision, or unbounded protection lease remains.

Cold destinations

The live source workspace is intentionally not retained as a loading cover for an already-reclaimed destination. Doing so would recreate the bug's timing-dependent dual-mounted view graph. A cold destination may expose the normal window background during its measured ws.switch.renderer-realization / ws.switch.first-frame interval; warm destinations stay on the immediate path.

The former cycle-hot 220 ms mount-pair timer and its now-dead two-workspace budget are removed; rapid cycling uses the same single authoritative mount path as every other switch.

The required tagged dogfood after CI should include an intentionally reclaimed destination and use these intervals to judge that visual interval. This PR does not claim an unmeasured p99 bound, and it does not trade the warm-switch ownership fix for another timer or callback-gated mount.

Diagnostics

The intervals use the existing DynamicTracingSignposts / OSSignposter path. Names and messages are not constructed unless a tracing tool is recording, and the phase names are readable directly in a Points of Interest trace.

Terminal realization ends on an observed drawable rather than when a nonblocking rebuild request is accepted. noteFirstFrame releases the frame observer and its notification-demand lease immediately, independent of source retirement, portal readiness, or interaction readiness; if no frame is produced, there is no per-frame notification work. The transaction owns at most one target-frame observer and one renderer lease per window; a later switch cancels and releases the previous transaction. The observer/demand pair also tears itself down on deallocation. Protected renderers are pinned inside, and count toward, the configured warm-renderer budget.

Regression coverage

The commits preserve a failing-test-before-fix boundary:

  1. 1a69a65869 adds the behavioral requirement that source retirement release switch-time renderer protection independently of presentation readiness.
  2. cfdaa39d10 makes mount reconciliation authoritative, rejects stale callbacks, rebases coalesced selection sources, and moves renderer-lease release to source retirement.
  3. e919239040 removes the dead live-source presentation path and adds request-scoped frame delivery and teardown coverage.
  4. 307bca49d9 makes protection self-expiring, reconciles authoritative stale deliveries, and covers presentation-target mismatch cancellation.
  5. 76a24646ae covers coalesced round trips and portal rebind presentation.
  6. 5f4b6931cb keeps selection side effects unconditional, makes disabled tracing allocation-free, and removes the obsolete source-cover policy.
  7. 8be3f7a6c0 makes frame observation RAII-owned, counts protected renderers inside the warm budget, and removes dead cycle-hot timer state.
  8. f1a50c3e79 splits the coordinator state model into one-type-per-file definitions.
  9. 5564407b5b makes the terminal portal registry the single source of presentation readiness.
  10. dacf39d15a adds the matching browser predicate and gates process-wide portal diagnostics to active, matching-window switches.

The test-only full CI run is pinned to 1a69a65869: https://github.com/manaflow-ai/cmux/actions/runs/30599009307

Additional focused coverage exercises portal/frame ordering, stale frame callbacks, reclaimed-renderer frame sequencing, rapid-switch cancellation, stale source callbacks, mismatch cancellation, owner expiration, deallocation cleanup, warm-budget accounting, and both idle and memory-pressure reclamation protection. I did not add a fake p99 UI assertion: intermittent SwiftUI/AppKit/Ghostty scheduling still requires a live trace, which is why permanent phase signposts are part of the fix.

Validation

  • swiftc -parse for all changed Swift and test files
  • ./scripts/check-pbxproj.sh
  • ./scripts/lint-pbxproj-test-wiring.sh
  • python3 scripts/check-test-determinism.py
  • python3 scripts/lint-stored-dispatch-work-items.py
  • bundled cmux policy check
  • git diff --check
  • Current-HEAD GitHub CI and bundled autoreview are the remaining gates

No local app build, local xcodebuild test, or XCUITest was run, per task constraints. Current main no longer contains .github/swift-file-length-budget.tsv or scripts/swift_file_length_budget.py; the new coordinator and its test file both remain below 500 lines, and .github/swift-warning-budget.tsv is unchanged.

Localization audit: no user-facing strings were added or changed, so Resources/Localizable.xcstrings and web message catalogs are unchanged.

Closes #9231

Closeout notes (2026-09-01)

  • The feature branch is rebased onto current main (2422b69c7c3555bd1efc5b37ffb32a7c619dc282) and the feature trees and messages are unchanged.
  • The current CLA policy could not resolve the historical cmux reload-cloud commit identity. I normalized author and committer metadata for the 49 feature commits to the PR opener austinywang; the test-before-fix commit sequence and all source trees remain intact. This is metadata-only and is required by the current CLA gate.
  • Renderer protection now ends only after source retirement and presentation readiness. For terminals that means both the portal and a post-selection drawable; browsers use portal readiness as their visual boundary.
  • Portal presentation notifications are deferred through a portal-owned, cancellable scheduler; hide, rebind, detach, and teardown cancel stale edges.
  • Deferred source unfocus is owned by WorkspaceHandoffRetirementGate and is flushed after the matching queued focus pass, so a coalesced callback cannot leave the old panel focused.
  • The coordinator remains in the app target because its production seam directly coordinates AppKit, WebKit, Ghostty, and the renderer controller. Extracting a package-only state machine would duplicate those lifecycle boundaries and is intentionally deferred as a separate architectural migration. The existing AppKit lifecycle test suite remains XCTest because its suite-level teardown owns global portal and window registries; moving one test to parallel Swift Testing would weaken that isolation.
  • Local macOS builds and tests were not run. Static parsing and repository checks passed; the required build and behavior verification are being run through the AWS and CI paths below.