cmux-tui: reads run no write; workspace.list and session.ping read only what they answer

Merged
LA
4d ago
+494-71

Summary

Under an 80-agent load the session snapshot was the workspace registry lock's longest holder: resource_api.rs:551 held it for 25.7 s in 60 s (fleet step 73e93323, server-stats locks from #28147). It also held mux.state for 23.3 s. Most of its calls were agents' CLI list calls, and two of its parts wrote to the database while holding the lock. Design rule 1 of the lock work (#27698) is that a read never writes under the lock.

  • A read runs no write.
    • The snapshot no longer deletes read marks of notifications outside the retained window. It skips them, and the registry open deletes them, because the runtime prune queue does not survive a restart.
    • It no longer publishes pending terminal directories (a full terminal.cwd commit) before it reads. Terminal output, resize, reconnect and every committed graph change still publish them.
  • One-collection reads read one collection.
    • workspace.list builds only the workspaces: workspace rows, the topology frame and workspace decorations, under the registry lock alone, without the state lock.
    • session.ping reads only the topology frame for its cursor.
    • One builder makes the workspace values for both the snapshot and the list, so the list equals the snapshot's workspaces (a test checks this).
  • Adoption publishes the shell's replayed directory (review of this PR, M1). The snapshot used to publish pending directories on its first read; that was the only step after a daemon restart that published an idle adopted shell's replayed directory (#12978). finish_terminal_adoption now publishes the pending directory and progress once the terminal is live, as a reconnect does. Red test: an_idle_shell_presents_its_reported_directory_after_its_host_adopts (red step 3ead7ba355c2aab83e7beeae at 72a5382b70f: "the adopted shell presents its launch directory, not its reported ..."; passes at the head). workspace.list also holds the registry connection for its whole read, so no late writer commit lands between its rows, frame and decorations (review L1).
  • #28159 review, Low 1 and Low 3.
    • lock_state_pinned settles the late answers that arrived when it pinned. An answer can still land after that settle, because the writer drops the connection and then sends its receipts. A pinned prepare now refuses while such an answer is unsettled, instead of building on state without the late commit. The next pin settles it.
    • A rename whose workspace left state after a durable commit raises the in-memory revisions before it reports the error.

An app terminal's first view also publishes the directory and progress its shell reported before the record existed (fresh review finding 1). Red test: the_first_view_publishes_a_directory_the_shell_reported_before_it (red step af09d2f0955b8bfff205602d at 6876e72646f).

Contains #28147 and #28159. Review only the commits after 352210fa9976. #27780 (Austin) touches the same lock. Owner: PL-daemon (session cmux-b6).

Testing

Red, at b870e5cfd36d (tests only):

  • Fleet step 0f68ce7b9cfd5e97c628bc68, cmux-tui-rust-check.sh test read_paths. Three tests FAILED for the stated reasons:
    • a_session_snapshot_read_runs_no_write: "session.snapshot wrote {resource_notification_reads}".
    • workspace_list_reads_only_workspace_tables: it read resource_screens, resource_panes, resource_tabs, resource_browsers, resource_terminals, terminal_hosts, the projections and session_journal.
    • session_ping_reads_only_the_topology_frame: the same tables.
  • opening_the_registry_drops_read_marks_outside_the_retained_window passed on the base too. It guards that the table stays bounded once the read stops deleting; it is not a red test.
  • Fleet step d5f923e12bcfe340a4c95b34: a_pinned_prepare_refuses_a_commit_that_landed_after_its_settle FAILED with "a pinned prepare built on a commit its state does not hold".

The tests record the tables each statement reads and writes through a SQLite authorizer on the registry connection.

Green:

Test fix, not a loosened test: named_server_status_stop_alias_and_durable_restart_preserve_topology.

  • It FAILED on this branch (step 23b44075, at d1457b2e06d6): server reload-config exited 3 with "the local server is still starting". It PASSED on the base (step 6317235aaefaacf34d409974).
  • Cause: the test sent lifecycle commands right after the socket accepted. The server accepts them only once it is lifecycle-ready (Mux::mark_server_lifecycle_ready). The test passed only while the workspace list before them was slow, and this PR makes that list fast.
  • The test now waits, after each start, on the product's own lifecycle_ready signal in identify. That wait is level-triggered (R5) on the product's signal, not a sleep for a fixed time: it reads identify every 25 ms until lifecycle_ready is true, bounded at 15 s, and every assertion is unchanged.
  • The product gap, a lifecycle command right after start fails instead of waiting on that signal, is in the handoff followups, owned by B2. It is not part of this PR.

Observed / Not run

  • Base reds (they fail on the base 352210fa9976 with the same assertion; this PR does not touch them):

    • flow_trace_bind::the_daemon_bind_path_starts_flow_marks ("socket path parent is not a directory: /tmp"; base step d73fc2c9).
    • replace_unresponsive_owner::ensure_reports_a_frozen_owner_unavailable_without_waiting_out_the_restore_bound ("not reported unavailable"; base step db1a43e8).
    • The CLI noun-first path test (#28119) and the acpmux hermetic guard (#28150).

  • The first green (23b44075) also had 9 acpmux reds. This PR cannot reach them: acpmux does not depend on cmux-tui-core. They are listed for a fixer, not cleared as flaky.

  • Not run: Austin's-day numbers, which the harness owner runs and posts. Perf claim: none until then. My diagnostic bench ranked this holder first; it is not the merge evidence.

  • R9 row R-L3 fails, owned by B2 (lead decision 2026-10-30). Both refusals in drain_unsettled reach clients as an untyped operation_failed with no busy code and no retry_after_ms:

    • workspace_registry/registry_intent.rs:244: "an earlier session journal registry commit is still unanswered; this request is retryable: retry now". This one exists on the base.
    • registry_intent.rs:251: "an earlier session journal registry commit landed after this request read state; this request is retryable: retry now". New in this PR. The next pin settles the late answer, so an immediate retry succeeds.

    The typed busy error with retry_after_ms is B2's next PR (pa-b2, after #28041): it adds a retryable registry.busy catalog error with retry_after_ms, and the CLI and hooks back off with jitter. pa-b2 has accepted both sites. Until it opens, the tracking entry is "NEXT PR after #28041, owner pa-b2" in the lead's followups. Until then, both messages say the request is retryable now.

Risk tier

The risk gate labels this PR risk:high: it adds a DELETE FROM (the new data deletion row). That line is prune_stale_notification_reads_at_open. It deletes the same rows the snapshot read deleted before this PR (read marks of notifications outside the retained window): the delete moved from every read to the registry open, and it deletes nothing new. The high tier needs an independent review receipt on the exact head and a green compile. A fresh review at the new head 481cab7733d9 follows its green; the review on 7093b67fd9ef no longer counts.

Performance

No perf numbers: this PR claims no measured speedup yet. My diagnostic bench ranked the snapshot reader first (step 73e93323); it is not merge evidence. The Austin's-day before and after numbers come from the harness owner and will be posted on this PR.

Changelog

Fixed: Agents' cmux workspace list calls no longer make the rest of cmux NEXT wait on a busy Mac.

Proof

No capture: daemon-only change, no UI.