PR #28147 - manaflow-ai/cmux

Summary

Under load, the cmux-tui session daemon could deadlock on its own mux state lock. When a registry commit outlives its 30 s journal receipt wait (a loaded Mac, #27698), its late answer is settled by the next request that calls Mux::lock_state_pinned, and that request holds the workspace registry and mux.state while it settles. The settle ran publish_resource_event. That function's publish_pending_terminal_directories step locks mux.state again. A debug build panics on the lock rank. A release build deadlocks the whole daemon (std Mutex), so every later request and every connection hangs.

The settle now publishes only the journal event. Subscribers still read the new head. A terminal directory that is still pending is published by that terminal's next output or the next committed graph change, the same as before the settle existed.

This PR also makes the daemon's two big locks measurable by call site:

server-stats with include:["locks"] reports the workspace registry lock and, for the first time, mux.state. For each lock it gives wait and hold histograms, the current holder, the last stall, the number of sites, and the top 20 sites by total hold and by p99 hold. Each site row has its hold and wait p99 and maximum. Each site keeps two 256-byte power-of-two histograms, updated under the site table lock that the release already takes, while the lock is still held. After a site's first acquisition this costs no allocation. The section is optional, so older SDK decoders keep working. Schema, bindings, api-compat surface and spec/commands.md are updated.
About 40 Mux lock wrappers get #[track_caller] (with_state, surface, resolve_resource_path, commit_state, commit_resource_mutation_plan, with_resource_projection, the resource effect helpers, lock_state_pinned and others). The statistics now name the request path that holds a lock, not the wrapper's own line.
cmux-tui/scripts/bench-daemon-latency.py --scenarios agents is a diagnostic fleet bench, not a CI step. It runs 80 fake terminal agents (the real hook helper plus a CLI call), terminal churn, and an app client (window saves, list-workspaces, session.snapshot, identify). It prints the app requests' latency and the lock sites.

#27780 (Austin) touches the same registry lock. The only file both PRs change is mux/resource_effects.rs, at different functions; I noted the overlap there.

Owner: PL-daemon (session cmux-b6), please review the daemon lock change.

Testing

Regression test: settling_a_late_registry_receipt_does_not_relock_state in cmux-tui-core/src/mux/tests/writer_lock_order.rs. It holds an indeterminate receipt, answers it, and pins state from another thread; the pin must return within 5 s and run the owed state update.

Red, fleet step a551489abaea4cfd8815ecf7: scripts/ci/cmux-tui-rust-check.sh test settling_a_late_registry_receipt_does_not_relock_state at 0f46ae7fa907 (test only, no fix). It FAILED with lock order violation: acquiring mux.state (MuxState) while this thread holds mux.state at mux/journal.rs:11, the deadlock path.
At the head d8f9ee99c02a (feat-cmux-next merged, plus the Rust panic ratchet fix), each fleet run is done (passed): settling_a_late_registry_receipt_does_not_relock_state (bd8f77630117abe7d6290e2d), lock_detail_reports_per_site_hold_and_wait_percentiles (e6b30baeb75b78399af62863), server_stats_include_reports_registry_and_state_lock_sites (197f5fab7581ddbebcc2fefb), clippy (a361a9dd9077dec93763fcd5). The PR's CI panic ratchet and crash ratchet pass.
Full run at the merge head c88bc3a8ff2d, fleet step aa49f608737c04b58e23478f: it compiled, fmt and clippy pass. Its 11 failures are base reds, not caused by this PR: the CLI noun-first path (#28119), the acpmux hermetic guard (#28150), and 9 new feat-cmux-next reds (agents_chat_tools x5, reply_in_place, a_daemon_whose_home_is_replaced_stops, integrity_check_scope, the_known_params_are_the_macos_hosts), whose fix feat-tip-reds has in progress. d8f9ee99c02a adds only the histogram change on top of it.
Earlier heads: green 52b69830 at 3b1cb411ebca and 78f9e8ec at 100b8e317c35 each failed on the two base reds only.
New tests for the statistics: lock_detail_reports_per_site_hold_and_wait_percentiles and server_stats_include_reports_registry_and_state_lock_sites.

Observed / Not run

Observed: the red failure above, and the green run. Earlier full runs on this branch also showed reds this PR cannot reach. They passed when run again alone and are listed for a fixer, not cleared as flaky: history_agent_fold::agent_history_save_does_not_wait_for_the_workspace_registry_lock failed at its precondition (line 109, before it takes any lock): a read-after-write gap between the agent-hook reply and the journal ring. B3 design item 3 owns it. Also apps_store::the_customize_folder_installs_itself_and_reloads_when_it_changes (step ba4f8faf). I also ran the agents bench at 6340c547fc53 (statistics only, no fix; fleet step 73e933233cd19e66b80674ea, aws-m4pro-9). It gave app requests p50 13 ms, p95 57 ms, p99 91 ms (1,868 requests, none over 2 s) and registry lock wait p99 82 ms. The longest registry holder is resource_api.rs:551, the snapshot readers: 25.7 s held in 60 s. These numbers rank the holders for the follow-up PRs; this PR claims no speedup.
Not run: the deadlock was not reproduced in a release build on a live loaded Mac. The release behavior follows from std Mutex relocking on one thread, which deadlocks or panics. Austin's-day numbers: pending harness owner. Perf claim: none; the statistics are instrumentation.

Performance

No perf numbers: the lock statistics are instrumentation and the fix removes a deadlock; this PR claims no speedup.

Austin's-day A/B: pending, in the pa-d queue (harness owner). It runs with an A/A-valid harness sha; the first A/A pass is https://github.com/manaflow-ai/cmux/pull/28077#issuecomment-6525676764. The A/B comment will be linked here.

Changelog

Fixed: The cmux NEXT session daemon no longer hangs when a busy Mac delays a save past its timeout.

Proof

No capture: daemon-only change, no UI.

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

Fixes the cmux-tui session daemon deadlocking when a late registry commit receipt is settled while the registry and state locks are held. The settle previously called publish_resource_event, which locks the state again — a self-deadlock in release builds and a lock-rank panic in debug. The settle now publishes only the journal event; a terminal directory still pending is published by the terminal's next output or the next committed graph change.

Bug Fixes

Regression test settling_a_late_registry_receipt_does_not_relock_state answers an indeterminate receipt and pins state from another thread; the pin must return within 5 s.
StateGuard now holds its MutexGuard directly, removing two expect() calls that tripped the crash ratchet.
The lock-site histogram uses bounds-checked bucket access and saturating arithmetic so it stays under the panic ratchet baseline.

New Features

server-stats with include:["locks"] now reports both the workspace registry lock and mux.state: wait/hold histograms, current holder, last stall, and the top 20 sites by total and p99 hold.
~40 Mux lock wrappers get #[track_caller], so the reported site names the request path holding the lock, not the wrapper's own line.
cmux-tui/scripts/bench-daemon-latency.py --scenarios agents is a diagnostic fleet bench with 80 fake terminal agents, terminal churn, and an app client; it prints app-request latency and lock sites. Not a CI step.
The locks section is optional, so older SDK decoders keep working; schema, bindings, and spec/commands.md are updated.