cmux-tui: one thread reaps every terminal host, not one per terminal (perf100)

LA
6d ago
+1187-7

perf100 tui lane, root cause 1 of N: a thread per terminal host just to reap it.

Each committed terminal host had its own terminal-host-reaper thread blocked in wait4 for the host's whole life. Lawrence's Chief daemon had 335 of them (1743 threads in total, 2026-10-20).

Change

  • terminal_host_runtime/sys/unix/host_reaper.rs: one terminal-host-reaper thread waits on kqueue EVFILT_PROC/NOTE_EXIT (Apple) or a pidfd in epoll (Linux) for every host, blocking with no timeout while no host exits. Statuses are collected with try_wait only. An exit event that arrives before the child is a zombie (XNU posts NOTE_EXIT first) re-arms: the kernel refuses an exiting process with ESRCH, and only then does the reaper wait (which returns at once). A stale event for a reused pid just re-arms. Other platforms, or a registration the kernel refuses, keep the old per-host thread.
  • tests/thread_budget.rs: behavior test at the socket boundary. A headless daemon with 24 cat terminals, each attached as the app attaches a visible terminal. Budget: 3.25 threads per terminal and 8 per attach connection. Every terminal echoes its input, and every host is reaped (gone, not a zombie) after the terminals end.
  • scripts/cmux-next/bench_tui_service.py: the lane's S0/S1/S3 bench for one cmux-tui binary (no app).

Evidence (rbx, Linux)

cargo nextest run -p cmux-tui --test thread_budget

  • red at 4221c48de48e (test only): daemon threads grew by 4.00 per terminal (38 -> 130); budget 3.25
  • green at 3725bf891a47 (fix): 3.00 per terminal

The stacked PR (shared executor) lowers this to 2.00. Full numbers: see the lanes PR.

Observed / Not run

  • Ran on rbx: clippy -D warnings (cmux-tui-core, cmux-tui), cargo nextest -p cmux-tui-core (2771 pass; 5 fail, the same 5 fail on base 78aaa620aa5e), thread_budget, idle_wakeups, terminal_resources.
  • macOS (kqueue path): hosted focused run cmux_next_daemon dispatched; result posted as a comment.
  • Not run: Windows (the change is Unix only).

Changelog

none

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Replaces the per-terminal-host reaper thread (one wait4 blocked for each host's whole life) with a single shared thread waiting on kqueue EVFILT_PROC/NOTE_EXIT on Apple and a pidfd in epoll on Linux for every host. Exit statuses are collected only with try_wait; early or stale events re-register the host, and only an ESRCH triggers a blocking wait that returns instantly. Platforms without these mechanisms, or registrations the kernel refuses, keep the old per-host thread. This removes one thread per terminal (335 were present in one Chief daemon); the shared reaper also survives a poisoned lock.

Testing and benchmarks

  • cmux-tui/tests/thread_budget.rs: a headless daemon with 24 cat terminals asserts thread growth stays within 3.25 per terminal and 8 per attach connection, and that every host is reaped (not a zombie) after the terminals close.
  • scripts/cmux-next/bench_tui_service.py: measures per-terminal threads, RSS, CPU, and context switches across S0 idle, S1 flood, and S3 scale scenarios; on Linux it counts CPU of hosts the daemon already reaped.