cmux-tui headless owner: the terminal reaper and the Chief brain supervisor no longer spin on a past deadline (idle CPU)

Merged
TE
3d ago
+168-18

The app's headless cmux-tui owner (cmux-tui --headless --session cmux-app-nightly) was seen at about 150% CPU for 40 minutes on an idle Mac, against a budget of no CPU at idle. Two background threads in that owner can run with a zero wait once a deadline is in the past and nothing moves it. Each is fixed here with a test that fails on base.

1. mux-terminal-reap: a reap attempt that leaves the terminal reapable re-runs at once

When a due terminal's reap came back Retained or Reaped, the reaper dropped its deadline (forget). If the next scan still listed the terminal, it re-added the deadline at now + grace - unplaced_for. Since #29476, unplaced_for is the durable time the terminal was first seen unplaced, which is already past the grace, so the new deadline was now.

The thread then waited zero (recv_timeout(0)) and ran the next pass at once, and kept doing so. Every pass costs:

  • a registry prune of keep and resume rows,
  • a SQLite sync_terminal_unplaced write,
  • a screen-archive capture,
  • a guarded close.

When several terminals are stuck this way, their attempts run in parallel on mux-terminal-work.

A terminal ends up stuck this way whenever the scan and the close guard disagree about whether it is placed. The scan matches a view's runtime id or the content's placements. The guard matches any surface with the same host identity.

Fix: ReapSchedule::tried postpones the deadline by grace.max(MIN_RETRY) after every outcome, as Attached and errors already did. A scan that no longer lists the terminal (closed, placed again, or kept) still drops the deadline in observe_unplaced, so terminals that were really reaped or re-placed behave as before.

2. chief-brains: a brain still typing after the idle stop is due leaves a past deadline

next_deadline offered the idle stop (last_activity + 30 min) for every running brain. on_time stops only a brain that is not typing. So a turn that kept its typing on for more than 30 minutes left a deadline in the past that nothing moved. The thread waited zero, did nothing, and ran again, holding one core at 100% until the turn ended.

Fix: a brain has no idle deadline while it is typing, which matches on_time. When the typing ends, handle sets last_activity, and the stop counts from that point.

Loops touched: stop conditions and cleanup

  • mux-terminal-reap:
    • Stop: the existing TerminalReaper::stop / drop (the stop flag plus closing the mailbox), and the thread ends when the Mux is gone.
    • Cleanup: terminal_reaper_events is cleared on exit (unchanged).
    • Waits now: at least MIN_RETRY (1 s) per stuck terminal, normally a full grace.
  • chief-brains supervisor:
    • Stop: Signal::Shutdown, or the inbox disconnecting (unchanged).
    • Waits now: until a signal, a backoff retry, or the idle stop of a brain that is not typing.
  • No new timers, observers or threads.

Tests (red on base)

  • terminal_reap::tests::terminal_reap_schedule_waits_after_an_attempt_that_left_the_terminal_reapable: one terminal unplaced for three graces, with a reap attempt that leaves it reapable. The test simulates the thread's waits for one second and counts the passes.
  • terminal_reap::tests::terminal_reap_schedule_drops_a_settled_terminal_once_it_is_no_longer_reapable
  • chief_brains::supervisor_idle_tests::a_brain_typing_past_the_idle_stop_leaves_no_past_deadline

Performance

metricbeforeafterdeltaharness or run link
reaper passes in 1 s, one terminal left reapable by its reap1000 (the test's cap; the wait is 0, so unbounded)1-99.9%terminal_reap_schedule_waits_after_an_attempt_that_left_the_terminal_reapable
reaper wait after that pass0 s30 s (the grace)+30 ssame test
chief-brains wait with a brain typing past 30 min idle0 s (one core spinning)none until its typing endsno spina_brain_typing_past_the_idle_stop_leaves_no_past_deadline
headless owner CPU at idle on the Mac that showed it~148% (40 min)to be read after the next NEXT nightlysample after nightly

Related: #33099 fixes the "cmux-tui stopped and could not start again" toast, not this cause. #33117 (open) adds a cap on the reap grace in the same file; the two do not conflict.

Agent session: agent session

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes two hot loops that spin the headless cmux-tui owner at ~150% CPU on idle: the terminal reaper and the Chief-brains supervisor can wait zero, doing nothing, when a deadline falls in the past and nothing moves it.

Bug Fixes

  • mux-terminal-reap: a reap attempt that leaves a terminal reapable now keeps a retry floor (the next deadline is the later of the scheduled one and the floor), so the terminal is retried after at least MIN_RETRY instead of being re-added as due at once. A terminal that a scan no longer lists drops its deadline and floor; one placed again and closed again gets a fresh grace from its new unplaced time.
  • chief-brains: a brain that is typing has no idle deadline (matching the idle stop in on_time), and the idle stop counts from when its typing ends, so a past deadline can no longer sit untouched and keep the thread spinning.
  • Adds four tests covering the waits; all fail on base.