acpmux: a failed Claude subagent says why and which agent resumes it, and its resume runs it again

Merged
TE
11d ago
+188-12

Leo, word for word (2026-10-26): "this needs to retry :(". His screenshot (private, not linked) showed two rows, "Subagent failed: Remote acpmux setup overview for Lawrence · 12m 3s" and "Subagent failed: Push relay for GCP lane bundles · 23m 45s". Both died on the SR2 gateway's "API Error: 503 no non-exhausted claude accounts available ... (error type server_error, HTTP 503 ...)".

subsession-card's ask (daemon half): "(a) A turn that fails on a transient upstream error (HTTP 503, 529 overloaded, "no non-exhausted ... accounts", "overloaded_error", or a connection reset or EOF to the gateway) retries by itself with backoff, then gives up. It resumes the same session and turn, never a new session. Emit a state on the session or turn that the webview can draw. Non-transient errors (auth, a bad request, a refused permission) never retry. (b) An RPC, a CLI command and an MCP tool that retry a failed session's last turn on demand. The webview's Retry button calls the RPC."

This is the first of two PRs for the subagent half (split to stay under ~800 lines). This one is translator-only: it gets the failure's error text and the agent id to the hub, and reports a SendMessage resume. #25234 (stacked on this one) adds the hub's retry.

What acpmux sees when a Claude subagent dies on that 503

Checked against hb-acpmux's own chat log (~/.acpmux/sessions/01a16c8c…, where five helpers died on the same 503 at 2026-10-26 00:38):

  • The update that carries it. For a background Agent subagent (both of Leo's rows), Claude Code writes system/task_notification {task_id, tool_use_id, status: "failed", summary: "Agent terminated early due to an API error: API Error: 503 no non-exhausted claude accounts available … (error type server_error, HTTP 503 …)"}. claude_stdio/inbound.rs subagent_task turns it into subagent_state_update {state: "failed"}. A foreground Agent call ends with the call's is_error tool result.
  • The error text did not reach the hub. The translator dropped summary. The hub saw {state: "failed"} and nothing more (record 71904 in that log).
  • The agent id did not reach the hub either. task_id is the id SendMessage resumes by: a8e1b68b93aac9d16 was both the failed task's id and the to of the parent's later manual SendMessage, and Claude Code answered {"resumedAgentId": "a8e1b68b93aac9d16"}. The translator kept it only in its internal subagent_tasks map.
  • The parent's state. No acpmux turn was running (session.turn() is None). Claude Code then starts a turn of its own to hand the notification to the model: system/init, then its result with "claude2: result with no client turn in flight" on stderr (records 71906–71946). #25234 waits that turn out.
  • A resume. SendMessage to the agent id restarts the same task: task_started {task_id: <agent id>, tool_use_id: <the SendMessage call>}. The subagent's own lines keep naming its first Agent call (records 72013–72064). Before this PR acpmux dropped the restart, because the SendMessage call is not a subagent it knows.

What changed

  • claude_stdio/inbound.rs: a failed subagent's end carries error (the notification's summary, or the failed Agent call's result, up to 600 chars) and agentId (the Claude task id; never for a workflow, which resumes by its run id). A later task_started for that agent id (SendMessage's resume) reports the same subagent running again and tracks it, so its next end ends it. task_tool finds a task line's Agent call by its tool_use_id when that is a known subagent, else by its task id (the resumed agent's lines name the SendMessage call); workflows.rs uses it too.
  • claude_stdio/mod.rs: failed_agents, at most 64 per chat; past that only the oldest is forgotten (a forgotten one's resume still streams under its own session; only its running again is not said here, and #25234 reads that resume from Claude Code's raw task_started itself).
  • subagents.rs: agentId and error pass through to _meta.acpmux.subagents.

No timers or observers are added.

Tests (red first)

Red at 2e25265 (tests only, https://github.com/manaflow-ai/cmux/actions/runs/40254651455): these three failed on the old translator.
Green at a9af7b3, the same change as the head 942d0f5 rebased onto #25232 (https://github.com/manaflow-ai/cmux/actions/runs/40258960219): fmt and clippy -D warnings clean, 113 lib tests (subagent claude_stdio) passed.

  • claude_stdio::subagent_tests::a_failed_agent_says_why_and_its_resume_runs_it_again
  • claude_stdio::subagent_tests::a_failed_foreground_agent_says_why
  • claude_stdio::subagent_tests::failed_agents_past_the_limit_forget_only_the_oldest
  • subagents::codex_tests::a_failed_subagent_carries_its_error_and_agent_id

Review

Codex gpt-6-astra (xhigh), read-only through the bare mirror. Round 2 BLOCKed A on one point: an agent forgotten past 64 failures was not seen when resumed. #25234 now reads that resume from the raw task_started itself, so A keeps its bound. Round 3: SHIP at 942d0f55443435237cea149a1baa543c8346c58a.

Reds: the only failing subcheck in "cmux-next checks" is release-compile-derived-data.test.sh (Swift bench flags). It and the other reds (merge tree, same-tree cmux-tui, release-notary, guard status, review-link and gate posts) fail the same way on merged #25232, so they are base reds, not this change.

No capture: daemon-only change, nothing drawn. The webview that draws a failed subagent's retry is #25226.

Agent session: big-red
🤖 Generated with Claude Code