PR #10564 - manaflow-ai/cmux

Summary

validate and commit one Ghostty app configuration without the native synchronous all-surface walk
compare canonical serialized effective config and bypass surface scheduling entirely for unchanged reloads
apply every focused or visible surface immediately, then drain offscreen surfaces one visit per deferred main-actor turn
coalesce superseded reloads while rolling back pending retry state before the replacement snapshot applies
acknowledge socket callers after validation and config commit instead of after visible/offscreen surface fanout
replace broad SwiftUI config invalidation with field-specific font events plus one scoped workspace/Dock/titlebar chrome identity
include theme background, foreground, and background opacity in that chrome identity so theme-only reloads refresh visible chrome at commit instead of after the deferred offscreen fanout (Bugbot finding, commits aab076e42e test / 4bd17117f9 fix)

Performance

Issue baseline on stable 0.64.22 at 126 workspaces / about 130 surfaces:

cmux reload-config: 4.405s

This branch on an isolated tagged Debug instance at exactly 130 workspaces / 130 terminal surfaces:

final samples: 0.14s, 0.10s, 0.10s
earlier five-sample run: 0.11s, 0.11s, 0.11s, 0.13s, 0.17s
a real changed-config sample returned in 0.14s; debug evidence showed offscreen apply finishing asynchronously at 0.32s total
unchanged samples logged contentChanged=0; the final implementation returns directly after commit and schedules no surface traversal

Wall-clock stall length is not a useful unit-test assertion, so deterministic scheduler/config identity behavior is covered separately and the runtime measurements above exercise the exact issue topology.

Architecture and concurrency

GhosttyApp remains the single @MainActor owner because Ghostty surface mutation and AppKit host refresh are main-thread-bound. TerminalConfigurationApplyScheduler accepts one immutable derived snapshot, applies the visible identity set, and owns the bounded/coalescing offscreen traversal.

The default yield seam intentionally uses the existing MainActorDeferredActionScheduler: it owns one cancellable task and yields before each bounded drain, while tests inject a manual scheduler. The package completion closure is an intentional synchronous integration seam rather than an I/O callback API: it reports that a main-actor C-surface traversal reached its fixed endpoint without introducing a task/continuation around every surface visit.

Verification

focused package suites: 8/8 passed
- scheduler: visible priority, per-turn bound, coalescing, shared snapshot, retry exhaustion, and retry-before-replacement rollback
- exact Ghostty serialization identity: equal/different finalized configs
- scoped chrome identity: tab font, unfocused overlay, divider, and host-layer mode
standalone SwiftPM still emits the repository's existing non-lib Ghostty binary-target diagnostic, but all selected Swift Testing cases execute and pass
tagged app build passed via CMUX_SKIP_ZIG_BUILD=1 ./scripts/reload.sh --tag issue-10557-reload-config-stall; the current merged HEAD also passed the Release build lane
python3 scripts/swift_warning_budget.py --log /tmp/cmux-reload-issue-10557-reload-config-stall.log: warning budget respected; .github/swift-warning-budget.tsv is untouched
final merged-HEAD CI: 18/18 jobs passed, including all app-host shards, release, build/lag, Swift packages, and aggregate gates (run)
pbxproj normalization, test-wiring, workspace-package-group, Package.resolved, and localization JSON guards passed
universal ReleaseFast GhosttyKit rebuilt with native Sentry disabled; the new API symbol is present in macOS, iOS device, and iOS simulator archives
localization audit: the reload feature adds no user-facing strings; the temporary merged-main journal diagnostic reconciliation was removed after its upstream fix landed, so this PR no longer changes the catalog

Final runtime verification (2026-09-08, fleet Mac)

Tagged Debug build of the fix commit 4bd17117f9 (built by reload-cloud.sh), pushed to leased fleet Mac cmux-austin-mini-0 (Apple M4, macOS 26.4, console session), Ghostty config isolated in the tagged bundle's own Application Support config.ghostty, driven over the tag debug socket with 127 workspaces / 130 terminal surfaces (126 created workspaces + 4 visible panes). A main-actor probe issued current_workspace (a v2MainSync hop) about every 10 ms throughout.

Reload	Socket ack	Surface fanout (debug log begin to end)	Probe RTT during fanout (median / p95 / max)	Probe RTT before / after (max)
unchanged config, 130 surfaces, `prioritized=4`	0.022 s	0.395 s, 130 per-surface refreshes	0.19 ms / 0.63 ms / 272 ms	62 ms / 0.7 ms
theme change (`background`/`foreground`), 130 surfaces	0.187 s	0.791 s	0.45 ms / 88 ms / 463 ms (includes a ~396 ms in-app window capture requested at ack)	23 ms / 0.8 ms

Issue baseline for the same topology on stable 0.64.22 was a 4.405 s main-thread block.

Theme-only reload, visible chrome timing (the Bugbot finding): in-app window captures via debug.window.screenshot before the reload, 0.40 s after the reload_config ack (fanout still running until +0.81 s), and after fanout end. Sampled pixels (R,G,B):

Region	before	at ack + 0.40 s (mid-fanout)	after fanout end
titlebar strip (600,14)	27,42,58	122,31,31	122,31,31
pane tab bar (400,42)	27,42,58	122,31,31	122,31,31
split divider (618,200)	27,42,58	122,31,31	122,31,31
terminal pane (400,300)	27,42,58	122,31,31	122,31,31

#1b2a3a = (27,42,58), #7a1f1f = (122,31,31): every visible chrome region already carried the new theme while 126 offscreen surfaces were still being applied, and the mid-fanout capture is pixel-identical to the final one.

Regression coverage for that fix: cmuxTests/TerminalConfigurationPresentationMetricsTests (18 parameterized cases) fails 3 cases before 4bd17117f9 and passes 18/18 after (SwiftPM harness on a fleet Mac); the hosted test-e2e.yml run on the merged HEAD 78c2095d8d passed (run 34209118550).

Debug-log attribution inside the fanout window: the per-surface applies are cheap; the largest main-thread gaps (55–138 ms) are sidebar row-projection rebuilds for the 127-row workspace list (5 rebuilds during the unchanged reload, 9 during the theme reload). That is the pre-existing sidebar path, not the surface apply, and is the remaining bounded cost at this scale (possible follow-up, out of scope here).

Trade-offs stated explicitly:

The runtime run used the build of 4bd17117f9; the later merge commit 78c2095d8d only brings main forward (no reload-path changes) and is covered by the CI build lanes and the hosted unit-test run above. The shared build queue on the dev Mac was held by other agents' local builds, so a rebuild of the merged HEAD was not waited for.
GitHub's API cannot upload images, so the captures are recorded here as sampled pixel values; the PNGs, results.json, debug-log windows, and the scenario script live at cmuxterm-hq/artifacts/verify-remote/issue-10557-reload-config-stall/2026-09-08T0945Z-run1-cmux-austin-mini-0/.
No timing was taken on the shared dev Mac (load average above 250); only the fleet Mac numbers are reported.

Closes https://github.com/manaflow-ai/cmux/issues/10557

Summary by CodeRabbit

New Features
- Configuration reloads now apply incrementally across terminal surfaces, prioritizing visible and focused surfaces.
- Added retries and safe handling when surfaces are unavailable or replaced during reloads.
- Configuration changes now distinguish terminal font, tab bar, sidebar, and chrome updates for more targeted refreshes.
- Added identity-based tracking to prevent stale configuration updates.
Bug Fixes
- Improved browser popup URL display behavior for blocked or failed navigations.
- Workspace appearance refreshes now occur immediately when returning to visibility.

[!NOTE]
Medium Risk
Touches the central Ghostty configuration reload path, surface lifecycle, and socket reload_config timing for all workspaces; behavior is heavily tested but regressions could show up as stale themes, missed surface updates, or changed reload acknowledgement timing.

Overview
Ghostty reload_config no longer walks every terminal surface synchronously on the main actor. After validating and committing app config (via ghostty_app_update_config_without_surface_propagation), visible/focused surfaces apply first and the rest drain through TerminalConfigurationApplyScheduler with coalescing, retries, and lifecycle-token lookups so replaced surfaces are not updated by stale work.

Socket and API semantics shift: reload_config can acknowledge on commit (commitCompletion) while propagation finishes later; overload returns busy instead of a false failure. ghosttyConfigDidReload is deferred until fan-out completes; SwiftUI chrome listens to narrower notifications (font, tab bar, sidebar, chrome identity) driven by TerminalConfigurationPresentationMetrics.

Reload-time surface creation is gated and keyed by surfaceID, with overflow recovery through the registry. Smaller follow-ons: cloud SSH bootstrap script moved to FreestyleInteractiveShellScript, Pi feed max timeout raised to 5s, UI tests ignore the SwiftUI bootstrap window, and Swift warning budget entries trimmed for fixed warnings.

^{Reviewed by Cursor Bugbot for commit 78c2095d8da0fa46403a006b2e588eb89995f24a. Bugbot is set up for automated code reviews on this repo. Configure here.}