Warn on runaway agent sessions: sidebar badge and acpmux ls WARN column

Merged
TE
6d ago
+856-25

Leo, 2026-10-30: "that visibility into [one agent host's] guys was (and is?) pretty bad i feel like we can make more improvements to acpmux/cmux-next"

Leo, on acpmux ls: "and acpmux ls should probably not be overly... verbose? i guess" / "to keep everything efficient... tho idk"

Part of HQ#1920, HQ#5226 and the HQ#5123 telemetry lane (row d: "Show unknown, never a synthesized zero. A metric must trigger an action."). Related: HQ#5266 (a revived session keeps its autocompact cap); this PR never warns on such a session's unknown cap.

A usage audit found four long-running Claude sessions with no autocompact cap carrying 290-330k of context per call and starting 67-90 Opus subagents. Nothing in acpmux or cmux NEXT showed it; finding it took a hand audit of ps lines and transcripts. This PR is the visible warning: a person sees a mark, not a log line.

What changes

  • acpmux decides the warnings, once. A session summary gets usageWarnings: [{rule, value, limit, text}], computed when the summary is read from its usageStats (the fields from the companion acpmux usage-stats PR) and the thresholds:

    • noAutocompactCap: a Claude session (family claude) whose usageStats.autocompact is none.
    • contextTokens: usageStats.contextTokens is above the limit.
    • subagentsPerHour: usageStats.subagents1h is above the limit.

    The warning list is empty for a closed session or one with no usageStats yet, so until the usage-stats PR lands nothing shows.

  • Unknown stays unknown. A count acpmux could not read is null, never 0. An autocompact of unknown (an adopted or revived chat whose launch acpmux never saw, or a setting it could not read) never raises or clears a warning. A value it lacks shows as ?, never 0.

  • Each warning names the action that answers it (action: {label, command, available}). The tooltip line ends with it, and nothing runs automatically:

    ruleactioncommand
    no autocompact capRotate onto a capped profilelane-place rotate NAME (coming: session restart keeps the launch settings, so it adds no cap)
    context above 400kCompact itacpmux send NAME /compact (Claude Code's own command; the Claude adapter already runs it as a prompt, see compact_ends_its_prompt_and_reports_the_compaction)
    subagents above 40/hOpen the session to see what it starts; cancel its turn to stop itacpmux session cancel NAME

  • One declared place for the thresholds: the new usageWarnings section of acpmux config.json, {"contextTokens": 400000, "subagentsPerHour": 40, "autocompactCap": true}. Those are the defaults (config/usage_warnings.rs), and 0 or false turns a rule off. Documented in the acpmux README and docs/acpmux-schema.json.

  • acpmux ls stays compact. Default ls gets a WARN column holding the single word warn per flagged row, shown only while some row has one; with no warnings the output is unchanged. The reasons and commands (ctx 412k: acpmux send NAME /compact; 90 sub/h: acpmux session cancel NAME) go to --wide, which the companion usage-stats PR adds. --json rows carry the whole usageWarnings. Computing the warning reads only the already-cached usageStats on the summary, and no transcript is read for it.

  • The cmux NEXT sidebar row gets a small warning triangle (exclamationmark.triangle.fill, theme attention color) in the trailing cluster beside the muted bell. It is its own view, not the working-state spinner. Its tooltip lists each warning's text, e.g. "90 subagents started in the last hour, above 40", and the row's accessibility label includes them. It is on for everyone: it does not depend on the sidebar.laneStatus / rowStatus / nowLine toggles. The Debug sidebar.laneFixture working sample now carries warnings, so captures show the badge.

How N was chosen (subagents per hour)

The rate is the peak count of Agent/Task tool calls in any sliding 60-minute window, measured per main transcript across every Claude transcript on this agent host (63 sessions, the last ~3 days):

peak subagents in any 60 minsessions
047
1-413
5-92
10-301 (the area coordinator, 27, which is intended fan-out)
> 300
the runaway deputies (audit)67-90

N = 40 sits above every ordinary session, with headroom over the busiest coordinator (27), and well below the runaway deputies. It is a config value, so it can move without a release.

The context limit of 400k is about where sr2's managed --autocompact window sits (380k-420k), so a session that compacts normally never reaches it. Five of the 63 sessions here peaked above 400k; all five are long-lived coordinators with no cap, which is the case this warning is for.

Tests

  • Rust hub/usage_warnings_tests.rs covers:
    • a runaway session breaks all three rules, with its numbers and text;
    • a capped ordinary lane breaks none;
    • the limits are exclusive;
    • only Claude is held to a cap;
    • a closed session, or one with no stats, gets no warnings;
    • a session with no cap but no context read yet still warns;
    • config overrides and 0/false turn rules off, and an all-default config writes nothing.
  • Rust cli/ls_table_tests.rs: warn_is_one_word_and_shows_only_while_a_row_has_one. The default ls shows one warn word; --wide shows each rule's short form and its commands.
  • Swift SidebarUsageWarningTests covers:
    • the lane parses usageWarnings;
    • the row carries its chats' warnings with laneStatus on and off;
    • duplicate lines show once;
    • the badge shows, with the tooltip and accessibility text;
    • a hovered row keeps the badge, at full opacity and left of its ⋯/× controls (theBadgeStaysBesideTheHoverControls). This test was committed red first, in 527848c8182, and the fix is 84e7b01d1d1. The first fleet capture showed the hover controls covering it.
    • a row with no warnings shows no badge.
  • Gallery AAUsageWarningGalleryTests (CMUX_TEST_REQUIRE_GUI=1, artifact usage-warning-gallery.tar): the real sidebar list drawn light and dark, before (no warnings) and after (the working chat breaks all three rules). Each still is also printed base64 to the log.
  • The sidebar owner's note (plans/cmux-next/sidebar-owner.md) logs the look change, and the commit carries a Sidebar-Look: trailer.

Observed / Not run

  • Observed:
    • The subagent-rate distribution above, from this host's transcripts.
    • Rust at 46be2bf (the acpmux code has not changed since): cargo clippy -p acpmux --all-targets --locked -- -D warnings is clean, and cargo test -p acpmux --lib -- usage_warn ls_table protocol_ir_is_current catalog_reload a_cell_without remote_summary peer_cache passes 32/32.
    • Independent Codex reviews (gpt-6-astra, high):
    • scripts/cmux-next/prepush-gate.py passes at the head.
    • Reds that fail the same way on base, per test (PR comments): the Rust panic ratchet (indexing_slicing 700 > 693, string_slice 98 > 94, the same at merge base e863077) and cmux-next checks. The latter's one counted red is String tables, with check-l10n.sh output the same at e863077; its sidebar-trailer annotation names three base commits, and this PR's range has 0.
    • The defaults guard's five "defaults changed" entries are fingerprints of functions that read Debug tunables. No default value changes, and the review receipt cleared it.
  • Not run: the Swift suites (SidebarUsageWarningTests, the gallery AAUsageWarningGalleryTests). CI's cmux-next swift test job waits on a fast-tier runner label that has started nothing, and this lane VM cannot build Swift.

Captures

Fleet captures from a build of the head, driven on a capture mini, checked by eye. The stills will be linked here once pr-media uploads work again.

  • (A) Rows, light, job 77c9fa09, head 84e7b01d against base e8630779. It ends on the frame with three workspaces and the Debug lane fixture on:
    • head: perf-sweep shows the red failed mark, flaky-ci the orange needs-you dot, and fleet-coordinator its working spinner with the warning triangle just left of it;
    • base: the same marks and no triangle.
  • (B) Tooltip on hover, dark, head 891176a8. It matches.
    • The row is hovered with ⋯/∨/× showing; the triangle stays at full opacity just left of ⋯, the pointer rests on it (⋯ is not highlighted), and no row card opens.
    • The tooltip lists all three lines with their commands.
    • In the still, the first 1-4 characters of each line are cut off at the left. That is only the capture cropping to the app window. The themed tooltip is its own borderless window (TooltipPanel), centered under the badge and clamped to the screen's visible frame (TooltipPresenter.origin, covered by ThemedTooltipTests). So a person sees it whole, and the crop is the roughly 700 pt wide tooltip running past the window's left edge.
    • An earlier run of B, job 8adf1a57 at 84e7b01d, showed no tooltip, because the Debug hover hook aimed at the badge's resting place, under ⋯. Fixed in 891176a8796.
  • Both runs were checked by eye by the capture lane. The stills are kept on the capture host (A: ~/.cache/pr-capture/84e7b01d/steps-ac9b6932cf/, B: ~/.cache/pr-capture/891176a8/steps-87a9bbecf9/) until pr-media uploads work.

No capture: the stills above exist (A at 84e7b01d, B at 891176a8, both matched by eye), but pr-media uploads are blocked for now; the image links will replace this line.

Performance

_acpmux/sessions latency (what acpmux ls calls), from the companion usage-stats PR's bench ls_latency: 30 fake-Claude chats that each ran a turn, 50 timed calls per mode, each on a new connection, on one 32-vCPU CI runner. Before is the companion branch b9c165d; after is the same tree plus this PR's daemon and config changes, so usageStats is present and every row's warnings are computed. Both were built once and run alternately (before, after, before, after); each figure is the mean of the two rounds. Command: cargo test -p acpmux --test hub_integration ls_latency -- --ignored --nocapture.

metricbeforeafterdeltaharness or run link
acpmux ls (no usage read) p5021.8 ms21.6 ms-0.2 ms (-1%)run 41012923271
acpmux ls (no usage read) p9528.0 ms27.9 ms-0.1 ms (-0%)run 41012923271
acpmux ls --wide / --json (usage read) p5021.1 ms21.3 ms+0.1 ms (+1%)run 41012923271
acpmux ls --wide / --json (usage read) p9526.6 ms27.2 ms+0.6 ms (+2%)run 41012923271

The deltas are within the spread between the two "before" rounds (p50 21.7 vs 22.0 ms; p95 25.9 vs 27.2 ms). Computing the warnings reads only the usageStats already on the summary: no transcript read, no new timer, no lock wait.

Changelog

  • Added: agent chats that run with no autocompact cap, a context over 400k tokens, or more than 40 subagents an hour show a warning badge on their sidebar row, whose tooltip says what to do, and a warn mark in acpmux ls.

Agent session: agent session

🤖 Generated with Claude Code