PR #23190 - manaflow-ai/cmux

Fixes https://github.com/manaflow-ai/cmux/issues/23087.

An idle headless cmux-tui owner (no terminal output, with or without clients) still woke on two timers for as long as it ran. Now neither runs while there is nothing to do, and the idle gate catches any daemon thread that wakes on a timer.

What was waking

Profile of an idle owner built from feat-cmux-next 6fa9a1281 (rbx, Linux, per-thread CPU and context switches from /proc/<pid>/task, plus the syscall each thread sat in): with a cat terminal, the app's streams attached and then closed, every thread was blocked except:

mux-agent-chat-notices: acpmux starts with the first agent chat, so on most machines it never runs, and this thread reconnected to its missing socket every 1-10 s forever.
cmux-journal-maintenance: a tick every 30 s with nothing to do.

The rest of the threads the issue names (mux-conn, terminal-host-reaper, mux-snapshot-attach-out, surface-*-host, journal-retention) made no switches in a quiet window: they block in read, wait4 or a condvar with no timeout.

The 5-16% owners the issue saw were not idle. The busiest one on the reporting Mac (cmux NIGHTLY at 828d36727, 19 agent terminals, app attached) spent its CPU in the session journal: over 30 s, cmux-journal-maintenance 1110 ms (sealing a backlog of terminal-output segments), mux-session-journal-writer 675 ms, mux-session-journal-fanout 159 ms, from about 27 terminal.output records a second. That build predates https://github.com/manaflow-ai/cmux/pull/22811 (output coalescing) and https://github.com/manaflow-ai/cmux/pull/22937 (terminal output leaves the journal), which remove that load on current feat-cmux-next. The other owners on that Mac were at 0.0-0.2%.

Change

Agent chat notices (mux/agent_chat_notices.rs): the thread waits on a file watch of the acpmux socket (cmux_config::watch, kernel events: inotify on Linux, FSEvents and a kqueue ancestor watch on macOS, including a socket directory that does not exist yet). The watch hints once when armed and on every create, replace or remove of the socket; each hint gets 3 connects (at once, 1 s and 2 s later), then the thread waits for the next hint. An ended connection reconnects the same way. If the watch cannot be armed, the old spaced retries remain.
Journal maintenance (mux/journal_maintenance.rs, mux/ledger_maintenance.rs): a tick reports whether work remains (ledger batches over their bound, a legacy auto-vacuum conversion, a checkpoint or seal, free pages for the incremental vacuum, a pause or an error). After a tick with none, the worker sleeps until Mux::publish_journal_event marks a journal change, or at most the hourly ledger scan interval, so the hourly scan still runs. Ticks stay at least one interval apart, and commits wake the worker only while it waits for a change. incremental_journal_vacuum now says whether free pages remain.

Test

cmux-tui/crates/cmux-tui/tests/idle_wakeups.rs (the existing idle gate at the socket boundary) now also follows each daemon thread over its quiet windows: a thread that runs in every window fails it. Its daemons get their own acpmux socket path (nothing serves it, as on most machines, and never the machine's own acpmux) and a 500 ms maintenance interval through the debug-only CMUX_TUI_TEST_JOURNAL_MAINTENANCE_INTERVAL_MS. Windows are 12 s, longer than the 10 s retry cap. After the windows, the test binds the acpmux socket and expects the daemon's initialize and _acpmux/watch within 3 s.

cargo nextest run --locked -p cmux-tui --test idle_wakeups on rbx (Linux):

red at fa76d8801 (test only, base sources): both idle tests fail, daemon threads that ran in each of the 3 windows of 12s: cmux-journal-ma, mux-agent-chat-; the daemon made 25-27 context switches per 12 s window.
green at 48248c41d (with the review fixes): 3 passed; the daemon made 0-1 context switches per 12 s window and connected 100 ms after the acpmux socket appeared, both with the acpmux folder present and with it created alongside the socket.

Dogfood on macOS (tagged build `23087-idle-owner-cpu`, fleet job 13fe4ea8ad31e8c2a44d4913)

The bundled cmux-tui of the tagged build at 48248c41d, against the bundled cmux-tui of a local tagged build at 1db785929 (an ancestor of the base with no changes to these files), each run headless with a cat terminal, an isolated HOME and an absent acpmux, per-thread CPU from proc_pidinfo over 60 s in 10 s slices:

	before	after
streams attached: CPU, wakeups	4.4 ms, 0.17/s	1.6 ms, 0.05/s
no clients: CPU, wakeups	1.6 ms, 0.18/s	0.2 ms, 0.03/s
`mux-agent-chat-notices` ran in	6 of 6 slices	0
`cmux-journal-maintenance` ran in	2 of 6 slices	1 of 6 (its first tick after setup), then 0
acpmux socket bound to connect	4.7 s	0.17 s

image-paste-expiry still rescans its folder every 30 s (2 of 6 slices in both); it is left as a rare bounded timer.

Observed / Not run

Observed: the red and green runs above and the macOS dogfood. On rbx at 48248c41d: cargo fmt --check clean, cargo clippy --locked -p cmux-tui-core -p cmux-tui --all-targets --no-deps -- -D warnings -A unused-qualifications clean (the two unused_qualifications errors are in mux/agent_reports.rs and server/tests/event_shape_tests.rs, untouched here and red at the base), and the full cmux-tui-core + cmux-tui suites: 5838 passed, 49 failed. 48 of those fail the same way at the base 6fa9a1281 on that host. The 49th, remote_link_state_dir::no_daemon_outlives_a_fixture, is load-sensitive: 40 parallel copies fail 25 and 20 of 40 at this head and 27 and 22 of 40 at the base, and it passes alone 3 of 3.

At the merged head 641229fad6b9: the same fmt, clippy and idle_wakeups + legacy_registry_maintenance runs (6/6), and 241 journal, ledger, checkpoint and projection tests in cmux-tui-core, all passed.

Not run: the idle test as a macOS fleet step. Step 016034126fd57f683fd5c5f9 (scripts/ci/cmux-tui-rust-check.sh test idle_wakeups at the red commit) selected 0 tests (idle_wakeups is the binary name, not a test-name substring) and exited 4, so it proves nothing; I cancelled the matching head step. The macOS dogfood above stands in for it. A symbolized sample of an idle owner on a Mac fleet host (perf is blocked on rbx, perf_event_paranoid=4, and the fleet runs only allowlisted measure scripts; per-thread names and CPU stand in for stacks). The frame producer (surface-*-frames) still wakes per output burst with no viewer attached; it is being reworked in https://github.com/manaflow-ai/cmux/pull/23113 and is not an idle wake.

No capture: no visible change (daemon threads only).

Changelog

Fixed: an idle cmux NEXT daemon no longer wakes every few seconds to look for the agent chat service or to run journal maintenance.

🤖 Generated with Claude Code