measure: idle CPU and wakeups of a build's app, acpmux and cmux-tui after N minutes idle

Merged
LA
2d ago
+747-0

Summary

On 2026-11-04 Lawrence's idle Mac had acpmux daemons at 18-52 % CPU each, with no user action (cmux-ad m9). There was no one command that answers "does this build rest when nobody uses it?". This PR adds one. It is a fleet measurement, not a CI gate.

  • scripts/measure/idle-cpu.sh (fleet step) runs scripts/cmux-next/idle_cpu.py:
    1. A fresh-home launch of a fleet build's exact app. The app zip comes from the controller store by sha256 and is checked. Nothing is built.
    2. Home must show in the first window.
    3. Optional: N agent chats whose first turn ended (the acpmux test agent, as bench_memory.py --fake-agent does).
    4. N minutes with no call into the app.
    5. Three measure windows: CPU %, wakeups/s and context switches/s for every process of the bundle (proc_pid_rusage at both ends of each window), one top cross-check, and sample PID 5 of each acpmux and cmux-tui daemon.
    6. Verdict lines: IDLE-CPU, IDLE-CPU-KIND and IDLE-CPU-VERDICT (budget per daemon process: 1 % CPU and 1 wakeup/s, median window).
  • scripts/cmux-next/idle-cpu-run.py --build JOB|--tag TAG [--mini cmuxNN] [--minutes 10] [--chats N] queues the step on a perf mini through the mini scheduler. It runs on the fleet host only, never on a person's Mac.
  • scripts/cmux-next/idle_cpu_verdict.py is the scheduler's verdict for that step.

DOGFOOD BENCH command (cmux-ad m9 item 3)

scripts/cmux-next/idle-cpu-bench.sh BUILD_JOB MINI [MINUTES] [OUT] (fleet host only) queues the step for that build on that mini, waits, and prints ONE JSON line last: {"app"|"acpmux"|"cmux_tui": {"n", "cpu_pct", "wakeups_per_s"}, "processes": [...], "busiest": {"kind", "pid", "cpu_pct", "top_frame"}, "verdict", "build", "job", "mini"}. Each value is the mean over the last 60 s after MINUTES idle (default 10). The top frame is the busiest non-waiting frame of sample on the busiest process. If nothing was measured, the last line is the reason in plain text. Service form (dogfood-bench/idle-cpu.cmd): bash <checkout>/scripts/cmux-next/idle-cpu-bench.sh {job} {mini} 10 {out}.

Repro options

  • --history N: seeds the fresh home with N Claude Code transcripts, N Codex rollouts and a Codex state DB whose writer was killed (its -wal and -shm stay). A fresh home did not show the idle loop.
  • --sample-each-window: after each window, one IDLE-CPU-WINDOW line per daemon and a sample of the busiest process. The sample runs outside the measured windows. Example: --windows 6 --window 600 gives 60 min with a sample every 10 min.

Testing

  • python3 -m py_compile on the Python files and bash -n on the shell scripts.
  • Local checks without an app: top_frame and bench_line on a synthetic sample report and synthetic rows; seed_history(…, 50) makes 50 Claude transcripts, 50 Codex rollouts and a Codex DB of 50 threads with its -wal and -shm left.
  • Not run yet: the controller refuses an unlisted scripts/measure script except at a feat-cmux-next commit. The first run (build 437f5a22, 30 min idle, 0 and 1 chats) is queued for when this lands.

Changelog

none

Proof

No capture: fleet measurement tooling, no UI.