PR #35400 - manaflow-ai/cmux

Summary

Stage S2c of the R-DURABILITY rewrite (design: handoff/rewrites/R-DURABILITY/DESIGN.md 3.3d). It follows #34901 (the acpmux event logs through one journal, store.journal, default off), which landed on feat-cmux-next as 6ad43f5d1bb; the base is now feat-cmux-next.

With store.journal on, a client is no longer told about a change before the change is durable:

Replies and acknowledged events wait for durability. Each connection gets an outbound gate (server/durable_gate.rs) between it and its transport. Every line it queues raises the gate to the store's last acknowledged LSN (Store::durability()), except the lines of agent output (a Stream record's event, delivered in a thread-local stream scope: DESIGN C4). The gate's task takes every queued line at once, waits (async, no thread blocked) until the journal's durable LSN reaches what they may show, then sends them in order: one wait per batch, as the journal's group commit is one sync per batch. A connection that heard nothing acknowledged never waits.
The daemon stops for a restart when the journal cannot go on. A failed log fsync freezes the checkpoint (#34901), and a journal writer that ends (a panic on its thread) closes the durable watch: each gate drops what it holds and closes its connection. In both cases the store asks the daemon to stop (Store::stop_reason); the daemon makes it a restart handoff (WebSocket clients get 1012) and exits with status 74 (EX_IOERR), which launchd KeepAlive and systemd Restart=always start again (the units prevent only 75). The next start's replay repairs the logs. DESIGN decision 9.8 (lead: keep it; no panic = "abort").
Without the journal, Store::durability() is None: no gate, and a send costs one Option check more.

I/O policy (lead rule 2026-11-04: a reply the person waits for never waits on a throttled disk sync). acpmux ran the whole daemon at IOPOL_UTILITY (qos.rs), and acknowledged replies now wait on the journal writer's sync. A thread's disk tier is the most throttled of its process's, its own and its QoS class's, so the writer cannot be raised alone. Measured on owned Macs (STATUS 11554; macOS 26.7 and 27.0.1): one acknowledged append to durable under another app's disk stress took 16.3-16.7 ms p50 and 37-45 ms p95 at utility, against 0.7-1.1 ms p50 and 1.5-1.8 ms p95 at the default tier. With the daemon (the real one, 16 chats streaming to two attached clients, plus another process's disk stress; scratch test tests/perf_ops.rs, never lands): after the Stream class fix (S2's ce8dcba5458), stress token_to_client (an agent's token to the attached client) p50/p95/max: journal on at the default tier 4.3/20.5/51 ms (fleet 2950e66f, aws-m4pro-4); journal off at the default tier 0.7/7.7/12.8 ms (b72720b0, aws-m4pro-9); journal off at utility, as feat today, 2.2/20.5/223 ms (c7b3649b, aws-m4pro-3). Acknowledged replies under stress, rename and draft_set p50/p95: journal on 21.2/25.8 and 15.0/19.5 ms; today 21.1/22.3 and 20.4/22.7 ms. Three hosts, one run each: a same-host repeat is a follow-up. Before the Stream fix, agent chunks were Ack records and the journal-on token p50 was 20.3 ms. So with store.journal on, the daemon keeps the default tier (qos::enter_daemon, which reads only that switch before any thread starts), and its background thread lowers itself: the checkpointer (qos::background_thread, a thread-scope utility tier for a thread that does nothing else). Without the journal nothing changes.

What changes

store/durability.rs (new): Durability { acked, durable }, streamed (moved from the journal store).
store/journal_store.rs: the acknowledged LSN (an atomic set in the append, before the hub publishes anything of the record), the durable watch fed by Journal::on_durable, and a guard owned by that listener that asks the daemon to stop when the writer ends while the store is not being dropped.
server/durable_gate.rs (new): the gate, the stream scope and install; server/mod.rs: Conn::gate, raised in send_replying, send_serialized, send_request_in_order and send_line; server/fanout.rs: the stream scope around a Stream record's delivery.
daemon/stop_causes.rs: store_asks_to_stop marks the stop and sets the restart handoff; daemon::exit_status, EXIT_REPAIR; cli/entry.rs exits with it.
Debug-build seams for the daemon-level test (release builds have none): ACPMUX_TEST_FAIL_LOG_SYNC (every log sync fails), ACPMUX_TEST_JOURNAL_CHECKPOINT_MS (the checkpoint period).
Moves, so server/mod.rs does not grow (it is over its size budget on the base): the outbound queue helpers to server/outbound.rs.

Review fixes (independent review FIX at d9938eb88d4, comment 6633213210)

finding	fix	regression test (red on the code before the fix)
F0: this PR's Linux clippy red, `unused import: super::*` at `qos_journal_tests.rs:6` (a macOS-only module)	the module is `cfg(all(test, target_os = "macos"))`	(the clippy step)
F1 (P2): a read could show a log line between its log write and its record (the append holds its lock across both, reads take none); the reply built from it went out with no wait	every journal-store read of the logs ends with a fence on the appends' lock (`Inner::fenced`), so a line the read saw has its record appended and its LSN acknowledged when the read returns; `Hub::scan_events`, the one read that sends as it reads (`session/load`'s replay), stops at the chat's `last_seq` read first	`a_read_that_shows_a_line_covers_its_record` (red: "a read showed line 2 before its record was appended (acked 1, then 2)", fleet 2eeae197); `a_replay_never_sends_a_line_before_its_record` (red: "the replay sent line 2 before its record: [1, 2]", fleet d0c6b406)
F2 (P2): a journal writer that keeps failing (it retries without end) held every reply on every connection with no limit	a connection's gate waits at most 10 s (`STALL_LIMIT`); past it, `Durability::stalled` asks the daemon to stop for a restart	`a_journal_that_syncs_nothing_asks_the_daemon_to_stop` (red: fleet e0ccde6e)
`threading-model.md` not updated	a row for the acpmux event journal's writer, checkpointer and per-connection gate, with their bounds	(doc)

The P3 items F3-F8 are follow-ups (Known gaps).

Known gaps

A change the hub makes in memory before its record is appended (a rename sets the chat's name, then appends renamed) can be seen by another connection's reply in that window. The reply then waits only for what was acknowledged before it, not for that record. S3 does not close it (DESIGN.md 3.3d, decided 2026-11-04): it needs every metadata write site to append before the change is visible; the window is microseconds on one thread, and a crash must follow within one group commit.
Review P3 items, follow-ups: F3 the stream scope also covers a watcher's summary for available_commands_update, built at delivery time (it can show a later acknowledged change unraised); F4 a tailnet device can get a close without 1012 when the writer ends; F5 qos::enter_daemon(true) sets nothing, so a daemon a utility-tier process starts stays at utility, and with the journal on the daemon's children run at the default tier; F6 retention holds a chat's append lock while it waits for the utility-tier checkpointer; F7 acknowledgements to the agent host still go out before durability (E12, stage S4); F8 a failed open logs a false "writer ended" error; no test covers the "writer ended" path.

Testing

Red on the code before each fix, green after (fleet steps, scripts/ci/cmux-tui-rust-check.sh):

server::durable_gate_tests (a connection through serve_connection on a journal store whose simulated sync takes 400 ms):
- an_acked_change_is_answered_once_its_record_is_durable: red at 2abbb1a0a4a (fleet 783893c4, hq run 41797031252): the rename was answered after 1.1 ms, before its record could be durable; also alone.
- agent_output_is_not_held_behind_another_chats_acked_change: green there and after (it guards the fix against holding agent output).
tests/daemon_restart_close.rs a_failed_log_sync_stops_the_daemon_for_a_restart (lead rule: the real daemon, a failed fsync through the hook, the restart exit and a typed client error, not a hang): red at 80c5fb2b7f8 (fleet 41f2d037, hq run 41804640229): the daemon exited 0 (wanted 74), while the 1012 close and the end of the connection already passed; the stricter version (the request in flight through acpmux::client::Client, the CLI's typed and retryable daemon_closed) is red at 31d276386b1 (fleet b6ef221a, hq run 41806243685): the 1012 close and the typed, retryable daemon_closed passed, and the daemon exited 0 (wanted 74).

Green at the head eb24c40e1dfe23615f170567d9c46dddb7f71371 (the review fixes; feat-cmux-next merged after #34901 landed: 9cf1425a7d5 merges 3dcd5db58a3 with S2c's side for the S2 files, 3969cb049b1 merges 35a1a909a11 cleanly):

fleet de2f31a2 (Linux, hq run 41852390302, at 3969cb049b1, the same code but the stall test): all store::: clippy pass, 318 of 318 tests. Its fmt step is red only on 7 acpmux files this PR does not change (agent_host/host.rs, claude_stdio/inbound.rs, cli/peer_status.rs, hub/admission.rs, subsession_mcp.rs, tests/host_lifetime.rs, tests/peer_status.rs): a base red on feat (from #33676 and direct pushes; a base-red PR fixes it).
The stall test, fixed so the journal ends (eb24c40e1df): red on the code before the F2 fix (fleet 92cff3a1: "a reply waited past the stall limit and the store did not ask to stop", in 3 s, no hang).
At eb24c40e1df on owned Macs: server:: 1576 of 1576, the three new gate tests included (fleet 51b2adb4, aws-m4pro-7); the restart test a_failed_log_sync_stops_the_daemon_for_a_restart (89fde7f4); daemon 833 of 834 (0033a5b0), the one failure being the base red acpmux_daemon_idle_has_no_periodic_wakeups (it also fails on the merge base alone, fleet 6c027917); sigterm 16 of 17 (10e278d9): daemon_binds_before_login_env_reports_ready_and_stops_on_sigterm said "ready after 2.10 s" against its 2 s limit on a loaded host; it passed alone at this head (7a9a3e38) and on feat d17e9736d91 (5d0e2ed2), and it failed on feat alone on another loaded host (the #34901 merge review, 6.85 s): a timing flake, not this PR.

Observed / Not run

Observed: the fleet steps above (Linux: Blacksmith 8-vCPU Ubuntu 24.04; macOS: owned Macs aws-m4pro-*).
Base reds, not in this PR's files: clippy type_complexity at cmux-tui/crates/cmux-tui/tests/link_push_bench.rs:59 (from #34538); acpmux::daemon_lifecycle a_daemon_whose_home_is_removed_stops (fails on feat d422b94b60d alone, owner BASE RED ACPMUX); server::fs_wire::tests::a_deflate_write_stream_over_its_decoded_size_leaves_no_file and acpmux_daemon_idle_has_no_periodic_wakeups (above).
Not run: a dogfood daemon with store.journal on (the switch is off by default).

Changelog

none

Proof

No capture: a daemon storage change behind a default-off setting, with no UI.