acpmux: the person's new chat never fails on a slow host; agents never take the person's warm agent

Merged
TE
6d ago
+324-6

Summary

Leo, 2026-10-31: "I TSHOULDN'T FUCKING MATTER IF MY COMPUTER IS LOADED" / "I SHOULD BE ALWAYS ABLE TO SATRT NEW FUCKGN CHATS".

On a host at load ~390 the daemon logged pooled session failed to start: no harness session within 90s harness=claude three times (04:36, 04:37, 04:56Z), and the person's new chats from the app failed. Both the pooled start and a cold start end at the 90 s agentStartTimeoutMs wall clock, whether or not the harness is still working on its start.

This PR covers items 3 and 4 of the ask, the smallest change that stops the failures:

  1. The person's start has no deadline. A chat the person starts (NewRequest::person: this host's app pane or a connection that proved it is the person, the same signal admission already uses, now one function person_asks) or writes to (session/prompt on such a connection) is marked Session::person. Its cold start waits as long as the start runs. Every 10 s (sooner when agentStartTimeoutMs is shorter) its log gets agent_start_waiting {waitedMs}, so the app can show "starting, this computer is busy". The daemon log warns once when the wait passes the budget. Only a real error ends the start: the harness exits, the spawn fails, or a stop or close kills the child. Starts by agents and lanes keep agentStartTimeoutMs, so an agent that never answers still frees its session (cx-m5up).
  2. Agents never take the person's warm agent. A create from an agent session (starter: the caller token the connection proved) neither claims a pooled agent nor warms one (pool_note_used). The pool stays reserved for the person's next chat. I found no other queue inside the daemon where a person's start waits behind lane starts: request slots are per connection, and the admission waits only on a name collision. The new test pins this down.

Not in this PR:

  • Item 1 (the person's harness at high priority, lanes at background): lawrencecchen's open #28923 rewrites the same CPU-class path (cpu_class.rs, ensure_child, pool/start.rs) so that every harness starts at normal class and is lowered after its start. Item 1 builds on that (the person's harness is then never lowered) once it lands, so this PR does not touch the class.
  • Item 2 (one warm harness per common harness). The pool and typing prewarm already exist; that follow-up keeps a bounded entry ready.
  • The pool's own 90 s START_BUDGET stays. A pooled start that misses it costs only the warm entry, and the chat then starts cold, which now waits.

Owner: PL-acpmux (hub/lifecycle.rs, hub/pool, server/requests.rs).

Testing

tests/hub_integration/lifecycle_fixes.rs, against the fake agent's FAKE_START_GATE (an agent that does not answer its start until a file exists):

  • the_persons_slow_start_keeps_waiting_past_the_start_budget_and_says_so (red on base: agent_start did not finish within 300ms): a person's session/new with a 300 ms budget is still waiting after three agent_start_waiting records, and it succeeds once the gate opens.
  • the_persons_start_fails_when_its_agent_exits: FAKE_DIE_AT_START still fails the person's start at once, and not with a deadline.
  • a_stop_ends_the_persons_start_that_still_waits: _acpmux/kill ends a person's start that is still waiting.
  • the_persons_start_never_waits_behind_lanes_that_are_stuck_starting: four agent connections' starts are stuck at their gate, and the person's chat on another harness starts at once (green on base too: this pins the absence of a queue).
  • an_agent_that_never_answers_its_start_fails_with_a_deadline (cx-m5up) now creates from an agent's connection and still fails at its deadline.

Results at c5ce9fee7280 (merge of feat-cmux-next, #28923 included): lifecycle_fixes 13/13 (the 4 new tests and cx-m5up pass), session_pool 34/34, clippy prints no warnings or errors (run 41261091912). At 36209a6f the same passed (run 41174618186). The first run, at 0b6e06225a, aborted with a stack overflow in an_idle_session_harness_exits_attached_or_not_and_resumes_on_the_next_prompt (debug-build stack). Boxing the cold start fixed it, and #28923 boxes it too.

After the merge, a person's start runs inside #28923's retry loop: it has no deadline, so it retries once only after a real error at background CPU class. Starts by agents and lanes keep #28923's loop unchanged.

Performance

metricbeforeafterdeltaharness or run link
failed starts, 24 at once per kind, 2 s budget, 64 busy loops on 16 CPUsagent path (old behaviour): 0/24person path: 0/24nonestarts_under_load (ignored test), run 41174618186
start latency p50 / max, same runagent: 435 / 536 msperson: 448 / 606 ms+13 / +70 ms (noise)same

This synthetic CPU load did not push the stand-in agent past the budget, so the bench does not reproduce the failure on the loaded host: it shows no regression, not a measured fix. What shows the fix is the red/green test above: the person's start keeps waiting past several budgets and then succeeds, where base fails at the budget. A bench with heavier load (disk as well as CPU) on a fleet Mac comes with item 1.

Changelog

Fixed: On a busy computer, a new agent chat no longer fails after 90 seconds while its agent is still starting. It keeps starting, and the chat says it is still starting.

Agent session: agent session

šŸ¤– Generated with Claude Code