PR #28041 - manaflow-ai/cmux

Part of #27698 (pa-b2; rows 1-2 of the B9 limits audit, and the "one connection per agent" part of row 1).

Summary

This is a correctness fix: no hook event is lost, and every wait is bounded or ends on a signal. It claims no load or CPU gain. The Austin's-day A/B (below) shows no measured difference in refused connections, hook latency, total hook CPU or CLI latency. In that scenario cmux CLI processes held the 64 connections, not hook helpers.

On Austin's nightly-next, the session daemon refused 55,162 connections in about 2.5 hours. The hook child lost events when that happened:

Every hook call opened its own daemon connection, so 80 agents' hooks competed with every other client for the daemon's 64 slots.
A daemon at its limit writes its busy line (type: "refused", server.busy) and closes. The hook child took that line for "an invalid response envelope" and dropped the hook event.
After a connect or write failure, the child retried after std::thread::yield_now() with no sleep. It ran a core for its whole 4 s bound, then gave up and dropped the event.

Now:

One queue per agent, one connection per agent. Each hook child appends its request line to the agent's queue on disk: <runtime dir>/hook-spool/<socket>.<terminal>.jsonl, mode 0600 (hook_spool.rs). The child that takes the queue's drain lock sends the queue in provider order over one connection, and waits for each receipt. The drain lock is an flock and is released when the process exits. All other children return at once.
A busy daemon never costs an event. A line leaves the queue only when the daemon sends a receipt or a durable rejection. A busy line, a retryable answer or a dropped connection is retried through RetryBudget, with no deadline. The stop signals are the daemon's answer, or its socket going away (R-L5). If the session is gone, the lines stay queued for the agent's next hook, and the daemon answers a resend from its idempotency receipt.
A queue has an owner and an end (R6, R-L5), with no timer. A sender that reaches the end of its queue deletes the queue files, the drain lock first. Only a live daemon answers a line definitively (a receipt, or a durable rejection). A stopped, restarting or crashed daemon is not an answer: the daemon removes its socket on every owner stop, including upgrades and restarts, and a crashed daemon leaves its socket behind. The lines then stay queued, and the next hook of any agent of the same session sends them to the next owner. Other sessions' queues are left alone. The one definitive end is cmux session NAME reset-state: it removes the session's saved state, then deletes the session's queues and counts their events. server-stats with include: ["hook_queue"] reports queued_events (this session's queued lines, so a stuck queue is visible) and reset_dropped_events.
The new owner sends what waits. A daemon that serves its socket starts one thread that sends the queues a stop or crash left (hook_helper::resume in main.rs), so a restart does not wait for an agent's next hook. Queue file names are digests of the socket name and the terminal id, so two sessions never share a queue.
Unix only. The queue needs file identity (an inode check after each lock). Windows has none, so its hooks send directly, with the busy-line retry.
The queue lives only in a private directory. The socket's directory and hook-spool/ must belong to this user, with no group or other write permission. Files are opened with O_NOFOLLOW and mode 0600. Without a private directory, the child sends its line directly, so a missing queue never costs an event.
The daemon says when it refuses a parked connection (R-L3). While the owner restores, it parks up to 32 connections unread; at the cap it closed the oldest with no answer. It now writes the busy line first (restoring.rs, outside the slot lock), so a hook that already wrote its line retries it instead of taking the close for a line the daemon never answers.
A refused hook connect is never silent now (R-L5), on Unix with a private socket directory. The old helper exited 0 on a refused connect and dropped the event, so nobody logged the refusal. On Austin's day, about 99 % of the refused connects at 64/64 had no client log. Now:
- The busy line, a retryable answer, or a connection dropped before the line was written: the hook retries the same queued line with backoff. The event is not lost.
- No daemon (socket missing or refusing): the line stays queued, and server-stats hook_queue.queued_events counts it until a daemon answers.
- A line that a live daemon takes and never answers (it closes on it 3 times in a row with no busy line, or does not read or answer it within 5.5 min, which is past its own longest journal wait): the sender gives that line up, so the agent's later events go on, and it counts the line in hook_queue.undeliverable_events.
- reset-state drops the session's queued events, counts them in hook_queue.reset_dropped_events, and prints dropped_hook_events (or hook_queue_error).
- The daemon still counts every refusal it writes in connections.refused.
- Not covered: on non-Unix, or with a shared socket directory, there is no queue. Those hooks send directly and still retry a busy daemon, but an event with no daemon at all is lost and not counted, as before.
The provider's tool call is never held longer than before. The provider waits only until its line is on disk; with no private queue directory, until its line is written. HANDOFF_WAIT (5 s; 2 s for Codex SessionEnd) bounds that wait as a liveness bound. SOCKET_TIMEOUT is removed.
One retry policy (cmux-tui-util/src/backoff.rs, RetryBudget): full-jitter exponential backoff (100 ms doubling to 1 s), with the daemon's retry_after_ms added as a floor. Two forms: until(deadline) for a caller with a total budget, and new() for a caller that stops on a signal.
The inbox (one try, 1.5 s) now names a busy refusal, not "mismatched request id".
First commit (no behavior change): the journal append moves out of cmux-tui-hook.rs, which was 51 lines over its god-file baseline. It now meets the budget, and its baseline row is removed.

What is copied from Chromium

RetryBudget is net::BackoffEntry (net/base/backoff_entry.cc) with jitter_factor = 1. The delay is initial * 2^(n-1), clamped to the maximum, and the wait is drawn uniformly from [0, delay]. Two changes:

BackoffEntry takes the later of the backoff and the server's Retry-After (SetCustomReleaseTime). Clients refused together get the same retry-after, so taking the later of the two sends them back as one wave. Here, the jittered delay is added to retry-after.
A total budget is optional, and a wait that would end past it is not taken.

Not in this PR (next steps of pa-b2)

The daemon's busy line does not send retry_after_ms yet. The hook reads it already. The next PR adds it, computed from the daemon's own state, on top of #27719 (austinywang), which changes the same listen.rs refusal path. The CLI (cli/wire/retry.rs, fixed 200/600/1500 ms) moves to RetryBudget in the same PR.
Ordering under a refusal: the provider is released when its line is written. If the daemon refused that write unread, the retry keeps queue order, because it is the same sender.

Performance

No gain is claimed: this is a correctness fix. Austin's-day A/B by pa-d (harness aa7c09ebaed, A/A-valid; 5 ABBA rounds on cmux24; base 3ecd8020cb1, head 059ac19abe9, the same product code as 6012bb4d6fa apart from the 2 panic-ratchet lines and the test seam). Lower is better; the delta is head - base with its paired 95 % interval.

metric	before	after	delta	harness or run link
refused connections per second	9.76	9.27	-0.64 [-1.52, +0.24], no difference	A/B
daemon peak connections	64 of 64	64 of 64	0	A/B
agent hook p99	5,035 ms	4,851 ms	-71 ms [-393, +250], no difference	A/B
agent hook CPU, total per run	158.8 s	159.2 s	+0.35 s [-2.1, +2.8], no difference	A/B
agent hook CPU p99	14.67 ms	14.51 ms	-0.12 ms [-0.20, -0.04] (0.8 %)	A/B
person `cmux` CLI p99	3,615 ms	3,231 ms	-586 ms [-1,971, +799], no difference	A/B
acpmux new-session p99	22.2 s	91.3 s	+56.1 s [+16.6, +95.6]: the same failed opens, one wave hangs to 90 s instead of failing at 10 s; no link from this diff (see below)	split, code check
acpmux RSS growth per hour	-15 MB/h	+111 MB/h	+68 [+20, +117]	A/B
runs with a 90 s `agent_start` hang (15 rounds, final)	5 of 15	9 of 15	+4 runs; McNemar exact p = 0.29: no difference (the hang follows the host); every other metric over 15 rounds: no difference, refused/s +0.38 [-1.17, +1.93]	A/B x15

Testing

Red tests first, each in its own commit, then each fix. The tests are in cmux-tui/crates/cmux-tui/src/bin/cmux-tui-hook/journal_append_tests.rs and run against fake daemons:

Red 1 (5d1fe1fcd49d), fixed by 624c6965309c:
- a_refused_append_is_kept_until_the_daemon_admits_it
- hooks_refused_at_once_do_not_retry_in_lockstep: 16 agents are refused at the same moment. Each is delivered, each comes back no sooner than retry_after_ms, and the return times spread over at least 25 ms.
Red 2 (2bd98d326a9), fixed by 78a8f220117d (head):
- one_agents_hook_burst_uses_one_connection_in_order: 6 hooks of one terminal while the daemon holds its first answer. The base opens 6 connections, and the held event lands last.
- a_hook_event_outlives_a_busy_daemon: the daemon is busy for 4.5 s. The base gives up at 4 s and loses the event.
- Guards that pass on the base too: a_rejected_hook_event_does_not_hold_back_the_next, busy_refusals_are_retried_with_backoff_not_a_spin (the daemon's own busy line, with no retry_after_ms: 2 to 12 connections in 400 ms; the base dropped the event with "invalid response envelope"), and a_missing_session_socket_fails_at_once_and_keeps_the_event.
backoff::tests (cmux-tui-util): waits double to the cap with retry-after added, a wait past a budget is not taken, no-budget waits reset after a success, and 64 draws spread over the jitter window.
Red 3 (6b0fcb1d2a38), fixed by 54003d8f19fc: the first queue-lifecycle tests. Review then found that "socket gone" is no definitive end, so red 4 replaced the two tests that dropped a queue for that reason.
Red 4 (ab3bdd2334f7), fixed by a63aa5356d2c (head):
- a_stopped_daemon_keeps_the_queue_for_its_next_owner
- another_sessions_queue_is_left_to_its_session
- a_missing_session_socket_fails_at_once_and_keeps_the_event
- a_queued_event_releases_the_provider_at_once (a daemon that closes unread held the provider for HANDOFF_WAIT)
- a_shared_socket_directory_gets_no_queue_and_the_event_still_goes
- a_planted_symlink_is_not_followed
- a_hook_that_queues_while_the_queue_is_deleted_still_gets_sent: the queue was unlinked before its drain lock, which left a late hook's event with no sender. A test-only seam between the two unlinks makes the race reachable.
- with the fix: hook_queue::tests (reset-state drops and counts only its own session's queues; a shared directory gets no queue) and server_stats_include_reports_the_hook_queue. These test a new surface, so there is no earlier behavior that could be red.

Fleet steps (scripts/ci/cmux-tui-rust-check.sh, through proof-step.py):

Commit	Step	Expected	Result
5d1fe1fcd49d (red 1)	test journal_append	fail	failed as expected, 64dc7da72892f08868cf6467: "invalid response envelope"
2bd98d326a98 (red 2)	test journal_append	fail	failed as expected, a87c71bc4092b2c7fc539684: "the burst reached the daemon out of order"
6b0fcb1d2a38 (red 3)	test journal_append	fail	failed as expected, 1e36c02be0485df09f4c6db7: "the ended session's queue was kept"
e8d0e3e76099	all	pass	failed (mine): E0433 `tempfile` in cmux-tui-core tests, 2 unused qualifications. 8536eab2d835b33f390c1367. Fixed in a63aa5356d2c
53634ef112bc (red 4 proof: ab3bdd2334f7 with its parent's test build fixed)	test journal_append	fail	failed as expected, 4a8e6618 (see the PR comment)
8e476a818652 (red 5 proof: head with 3 fixes reverted)	test hook_queue / the_next_owner_sends	fail	failed as expected, 7a16d331 and c162151d
5a09468c61fc	all	pass	fmt and clippy passed; this PR's tests pass; others are base or other crates, a6b19209

Hosted focused runs (cmux-tui.yml):

Commit	Filter	Expected	Result
5d1fe1fcd49d (red 1)	`journal_append`	fail	failure as expected (Linux test job): 3 tests failed. Errors: "invalid response envelope", and "2642515 tries in 400ms". Run 41012836527
624c6965309c (fix 1)	`journal_append` / `backoff::tests`	pass	runs 41012695449 / 41012698277: `test (linux)` success for `backoff::tests`
2bd98d326a9 (red 2)	`journal_append`	fail	failure as expected (test linux), run 41016485345
78a8f220117d	`hook_helper` / `backoff`	pass	`test (linux)` success, runs 41016487955 / 41016490342

The hosted lint (linux) job fails at "rust-panic-ratchet: RED", the same on the red commit. It is a base finding, linked in a PR comment.

Compile-check: 5a09468c61fc passed (d485721c2935ef75244f6e94). Head b17d731fe653 (Windows: no queue): compile passed (f2f034b05cb436c64f99e14f). Its green 9ec64cabf25151aa6b905a96: fmt and clippy passed, and every test of this PR passed (41 PASS lines, no FAIL). Its failing tests are the two base reds and three tests that failed in the full run and passed alone, in crates this PR does not change (see the PR comment).

Spec: sdk-schema.json and commands.md (the sources) gain hook_queue. The bindings come from cmux-tui/bindings/codegen/generate.py --write. api-compat.surface.json is the snapshot of the cargo binary api-compat-check --write, which agents do not run on this VM. Its two new rows follow coverage.rs for an optional object with one required uint64, as the write_path rows do, and the PR's cmux-tui-spec job compares them with the generator's output.

P-AUSTIN A/B (15 paired ABBA rounds, final): https://github.com/manaflow-ai/cmux/pull/28041#issuecomment-6531295503. Harness sha 37b2e86471b (aa7c09ebaed plus a count of the 90 s hangs, acpmux log copies and round numbers; no file that measures changed, so A/A-valid with aa7c09ebaed: https://github.com/manaflow-ai/cmux/pull/28077#issuecomment-6525676764). Base 3ecd8020cb1, head 059ac19abe9 (later heads add only the test seam, the two panic-ratchet lines and the merge with feat-cmux-next). Result: no difference on any metric, refused per second +0.38 [-1.17, +1.93]. The 90 s agent_start hang is within noise: head 9 of 15 runs, base 5 of 15; McNemar exact p = 0.29, Fisher exact p = 0.27. It follows the host (cmux24 in both arms; on cmux19 the head never hung). The first 5 rounds (below) were a chance run on one host.

First 5 rounds (harness owner pa-d): https://github.com/manaflow-ai/cmux/pull/28041#issuecomment-6527503800. Harness sha aa7c09ebaed (A/A-valid; A/A: https://github.com/manaflow-ai/cmux/pull/28077#issuecomment-6525676764); summary by the reviewed summarize.py at e8349e8df0b. Product measured: 059ac19abe9 against its merge base 3ecd8020cb1, 5 ABBA rounds on cmux24, paired 95 % t-interval. Later heads add only the test seam (4f207e21469), the reset-state JSON build and saturating_add (fcacd42baab), and the merge with feat-cmux-next (6012bb4d6fa).

No difference: refused connections per second (-0.64 [-1.52, +0.24]), refused share, daemon peak connections (64 of 64), agent hook p99, total hook CPU per run, CLI p99, lock wait p99.
Better: agent hook CPU p99 -0.12 ms [-0.20, -0.04] (0.8 %); acpmux 10 s host-start time-outs -29.8 [-55.5, -4.1].
Worse: acpmux new-session p99 +56.1 s [+16.6, +95.6] (25 chat opens per head run fail on the 90 s agent_start time-out instead of the 10 s host-start time-out; the failure share does not change), and acpmux RSS growth +68 MB/h [+20, +117]. No link from this PR's code to acpmux agent start is found yet; the pa-b10 owner and this PR's owner check it.
In this scenario the 64 connections are held by cmux CLI processes (median 62-63 alive, 0 hook helpers), so the hook spool cannot lower the refusals here. So the PR claims no load gain. pa-d's per-run acpmux split (https://github.com/manaflow-ai/cmux/pull/28041#issuecomment-6527680275): the share of failed chat opens is the same in both arms. In each run one wave fails all 25 opens. On the head that wave hangs at the 90 s agent_start limit in 5 of 5 runs; on the base this occurs in 1 of 5 runs, and the other base runs fail it at the 10 s host start. The head adds no failures. The code check (https://github.com/manaflow-ai/cmux/pull/28041#issuecomment-6527710930) found no path from this diff to acpmux: acpmux does not link the changed crates, and its only daemon call (the 2 s launch-credential mint) fails fast with the busy line as it did with the close. The host-start cause is the harness exec's background CPU class (pa-b10's clamp PR).

Observed / Not run

Observed: prepush-gate.py reported no new violations. The god-file check passed on every changed Rust file. Red 1 failed on the hosted Linux job for the stated reasons.
Not run: the Windows test (no_hook_queue_without_file_identity, cfg(not(unix))); no fleet step runs Windows, so only the hosted full run can.
Known delay, not a loss: on macOS a live daemon with a full listen backlog refuses a connect; queued events then wait for the next hook or the next owner.
Not run: a live burst against a tagged build.
Observed (delta 059ac19abe95..4f207e21469, the crash-ratchet fix): verified on Linux only (Blacksmith overflow runners). Clippy -D warnings passed (step 7001ddc5372930e154fcbf5d, hq run 41077277684: clippy pass). test journal_append (step 4e9ad005defdc4122f026c77, hq run 41077610877): nextest Summary [5.288s] 35 tests run: 35 passed, 12698 skipped, including a_hook_that_queues_while_the_queue_is_deleted_still_gets_sent, which uses the changed seam. test hook (step 8f2e9756ed6e50afdc290998, hq run 41077651029): Summary [6.194s] 183 tests run: 183 passed, 12550 skipped. The crash ratchet's rust cmux-tui unwrap count is 429 again (431 at 059ac19abe95).
Fixed after review FIX at 4f207e21469 (comment 6526371650): the panic ratchet of "clippy (changed cmux-tui packages)" counted 3 hits this PR added (session_reset.rs:107-108 indexing_slicing, hook_spool.rs:105 arithmetic_side_effects). They were hidden at 059ac19abe95 because clippy stopped at the base launchd_refresh.rs:329 error before the ratchet step. fcacd42baab removes them; the baseline does not change.
Observed at fcacd42baab (Linux fleet, Blacksmith overflow): clippy -D warnings green (cb8a66174e3028b1eacf3a7d, hq run 41080562729); test journal_append Summary [4.938s] 35 tests run: 35 passed, 12698 skipped (55b90688171d4f99f6a8b4b0); test hook Summary [6.219s] 183 tests run: 183 passed, 12550 skipped (b3d3ee8a01edd54186bf337a); test session_reset_state Summary [0.129s] 13 tests run: 13 passed, 12720 skipped (ddb6b179959faa5392f960c8).
Merged origin/feat-cmux-next 17368d28c654 into the branch (6012bb4d6fa, a clean merge; generate.py --check, check-sdk-schema.py and check-resource-api-boundary.py pass).
Observed at 6012bb4d6fa (macOS, macos-27): test hook Summary [9.719s] 178 tests run: 178 passed, 12741 skipped (7814e05ee5935e539716865e, aws-m4pro-3); test session_reset_state Summary [0.682s] 13 tests run: 13 passed, 12906 skipped (c44a87239473e65bab5e1239, aws-m4pro-2); fmt, clippy and test journal_append (fb8745cdbcf3231af7557c1d, aws-m4pro-4): fmt pass, tests Summary [5.250s] 35 tests run: 35 passed, 12884 skipped (with a_hook_that_queues_while_the_queue_is_deleted_still_gets_sent), clippy red only on the base finding acpmux/tests/daemon_memory.rs:82 start_logged is never used (its only caller is #[cfg(target_os = "linux")]; from base #27991 by the merge; classified in comment 6527367112).
Running, started before the merge: macOS at 4f207e21469 (e38f13864e53c3a773970d85, f38c3149f2ccc0fbf04c8a61); 6012bb4d6fa supersedes them.
Panic ratchet: the controller refused .github/scripts/cmux-tui-clippy-changed.py --panic-ratchet as a fleet step ("invalid ci-step: script is not on the ci-step allowlist"). The PR's own CI "clippy (changed cmux-tui packages)" at 6012bb4d6fa (job 124149872633, pass) ran it: cmux-tui indexing_slicing 861 = baseline 861, arithmetic_side_effects 1188 = baseline 1188 (863 and 1189 before fcacd42baab). It warns acpmux: clippy::arithmetic_side_effects 1433 > baseline 1430 (warning level; acpmux is not in this PR; base, by the merge).

Changelog

Fixed: Agent hook events are no longer lost when the cmux NEXT session is busy or restarting; a hook now waits for a busy session with backoff, and an event that cannot be sent is counted in server-stats.