acpmux: a subagent a transient upstream error failed is resumed in its chat

Merged
TE
10d ago
+1535-4

Leo, word for word (2026-10-26): "this needs to retry :(". His screenshot (private, not linked) showed two rows, "Subagent failed: Remote acpmux setup overview for Lawrence · 12m 3s" and "Subagent failed: Push relay for GCP lane bundles · 23m 45s". Both died on the SR2 gateway's "API Error: 503 no non-exhausted claude accounts available ... (error type server_error, HTTP 503 ...)".

subsession-card's ask (daemon half): "(a) A turn that fails on a transient upstream error (HTTP 503, 529 overloaded, "no non-exhausted ... accounts", "overloaded_error", or a connection reset or EOF to the gateway) retries by itself with backoff, then gives up. It resumes the same session and turn, never a new session. Emit a state on the session or turn that the webview can draw. Non-transient errors (auth, a bad request, a refused permission) never retry. (b) An RPC, a CLI command and an MCP tool that retry a failed session's last turn on demand. The webview's Retry button calls the RPC."

This PR is the subagent half. The parent's turn did not fail, so the turn retry (hb-retry-turn's PRs) never sees these failures. It is split in two to stay under ~800 lines: #25264 (the translator: the failure's error and agentId reach the hub, and a SendMessage resume reports the subagent running again) lands first, and this PR, stacked on it, is the hub's retry. Its own diff is the last commit.

What acpmux sees when a Claude subagent dies on that 503

In #25264's body, checked against hb-acpmux's own chat log. In short: a background agent's failure is system/task_notification {status: "failed", task_id, summary}; task_id is the agent id SendMessage resumes by; the hub saw no turn running, and Claude Code then ran a turn of its own (system/init to result, "claude2: result with no client turn in flight") to hand the notification to the model; a resume is task_started {task_id: <agent id>, tool_use_id: <the SendMessage call>}.

What changed

  • Uses #25232's turn_errors::transient_cause (merged; this PR adds no copy).
  • hub/subagent_retry.rs (new):
    • When a top-level subagent ends failed with an agent id and a transient error, acpmux waits autoResumeDelaysMs (the same waits and per-chat stagger as the turn retry). It then sends the parent chat, in the same session, one prompt with _meta.provenance {kind: "subagent_retry", subagentId, agentId, attempt}. The prompt tells the agent to resume subagent "" (agent id X) with SendMessage to X, and to start a new subagent only when that resume fails. acpmux never starts a new session.
    • If acpmux's own turn is running, the prompt queues behind it and is never steered in. If Claude Code's own turn is running (Session::harness_turn, from system/init to result), or a turn continuation may still fire, the prompt waits for it to end.
    • It is skipped on Stop and close, when the subagent runs again (resumed) or ends, and when a new top-level subagent of the same name starts (replaced). Claude Code's raw task_started for the agent id counts as the resume too, so it is seen even when the translator forgot the agent. A prompt that already went out is taken back then: out of the queue, and refused if it would start later (Hub::start_turn checks Retries::start under the lock a skip takes). One that started already is the agent's to answer; its text says to do nothing if the subagent already runs again. Once a sent prompt's turn ends with the subagent still failed (the agent did not resume it), or the resume fails again, the next attempt waits; after the configured attempts it is given up (exhausted).
  • Contract state, unchanged names: while a retry waits, the subagent's entry carries retry: {attempt, of, reason, nextAt} and retryable: true. The same subagent_state_update is logged again when the retry fires (retry: null), skips or is exhausted. retryable says whether a person's Retry can work: true while the agent id is known and the chat is open, after a Stop and after the automatic attempts are used up too (then retry: null, state failed); false once the chat closed or the subagent runs again, ended, was replaced, or was forgotten. A chat keeps at most 32 failed subagents no retry waits for; past that the oldest is forgotten. Each step also logs a mux subagent_retry {subagentId, agentId, reason, outcome: scheduled|fired|skipped|exhausted, attempt, of, nextAt?, promptId?, skipped?}.
  • Not here: the on-demand retry (_acpmux/retry {sessionId, subagentId}, its CLI and MCP tool). hb-retry-turn owns that handler (its PR 2); the subagent branch goes on top of it once it lands, never as a second handler. One click is one attempt; it never re-arms the automatic schedule, and two clicks while one attempt runs answer retry.busy.

On a usage limit: a subagent that hit a limit is not retried here. Another account solves that, and hopping accounts on a 15-second capacity blip would not recover sooner.

Tests (red first)

Red at 2e25265 (tests only, https://github.com/manaflow-ai/cmux/actions/runs/40254651455): a_subagent_a_503_failed_is_resumed_in_its_chat and the_retry_waits_for_claudes_own_turn failed (no retry); the bad-request test passed, as it should on base.

Green at the head 7694b99 (remote-test on the fast-checks job; the last two runs: https://github.com/manaflow-ai/cmux/actions/runs/40270466189 at aa4f064 and https://github.com/manaflow-ai/cmux/actions/runs/40276386902 at 7694b99): fmt and clippy -D warnings clean, 132 lib tests (subagent turn_errors auto_resume claude_stdio) and 117 hub_integration tests (subagent_retry auto_resume stop claude queue) passed.

  • hub_integration::subagent_retry uses fake_claude.py's new "agent-503" scenario (a background agent fails on the 503 after the turn, then Claude Code runs its own turn for the notification):
    • a_subagent_a_503_failed_is_resumed_in_its_chat: scheduled, then the prompt with its provenance, then the SendMessage resume makes the same subagent running, and nothing more fires. The waiting entry carries retry {attempt, of, reason: "capacity", nextAt} and retryable: true.
    • the_retry_waits_for_claudes_own_turn: it does not fire while Claude Code's own turn runs.
    • the_retry_queues_behind_a_running_turn: a person's turn runs when it fires; the prompt waits in the queue and runs after it.
    • stop_skips_the_wait (retryable: true) and close_skips_the_wait (retryable: false): skipped, never fired.
    • a_resume_of_a_forgotten_agent_ends_its_retry: 64 other agents fail first, so the translator forgot the agent; the raw task_started still ends its retry.
    • close_after_stop_or_giving_up_is_not_retryable, failures_past_the_limit_are_forgotten_after_a_stop_or_giving_up.
    • a_forgotten_agent_that_keeps_failing_is_given_up_after_the_attempts, a_forgotten_agent_that_completes_after_its_resume_ends_completed.
    • Unit: a_skipped_retry_prompt_never_starts (also: the same prompt id in another chat starts), failed_subagents_past_the_limit_forget_only_the_oldest.
    • In-crate race tests (hub/subagent_retry_tests.rs, which drive the observer directly so each interleaving is fixed): Stop before the first wait is planned, a stopped wait woken by hand, a failure while the chat closes, close after a planned wait, a retry prompt waiting out Claude Code's own turn at its start, one skipped while it waits there, and a raw failure while the chat closes.
    • a_subagent_a_dropped_connection_failed_is_resumed: reason transport.
    • a_subagent_that_keeps_failing_is_given_up_after_the_attempts: scheduled, fired, scheduled, fired, exhausted; then state failed, retry: null, retryable: true.
    • a_subagent_a_bad_request_failed_is_not_retried.

Timers and observers, with their stop conditions

  • One task per waiting retry. It sleeps autoResumeDelaysMs[attempt] (at most 1 h each, 4 attempts) or until woken. It ends when the retry fires, or is skipped by Stop, close, resume, replacement or the subagent's end (each wakes it), or when the attempts are used up.
  • After its wait, while Claude Code's own turn or a continuation runs, the task waits on the hub's event broadcast and rechecks two flags per record. It stops at that turn's end, or at Stop, close or a resume (each wakes it). No polling. A turn acpmux runs is not waited on: the prompt queues behind it.
  • After the prompt goes out, the task ends when prompt_with returns (its turn ended); then it plans the next attempt or gives up.
  • hold_retry_start (at a retry prompt's start, holding the turn slot) waits on the event broadcast only while Claude Code's own turn runs. It ends at that turn's result, the host's exit, a skip or close. Any other prompt passes after one lock and one lookup.
  • observe_task_line runs on Claude Code's raw task_started and task_notification lines: one lock and one scan of the chat's subagents.
  • observe_subagents runs only on live updates that carry _meta.acpmux.subagents: a lock and a map lookup per entry, nothing per token. A chat's entries go at close.

Review

Codex gpt-6-astra (xhigh), read-only through the bare mirror. Round 1 (at 7deb701) BLOCKed on six points: the fire/Stop race, duplicate on-demand retries, exhaustion not terminal, clearing all failed agents past 64, the harness-turn window, and a 1 h busy timeout. All addressed: a fired wait stays tracked until its turn ends and a skip takes its prompt back from the queue; the on-demand retry moved out; exhaustion is final with retryable: false; the oldest failed agent is evicted only; BUSY_WAIT is gone. The harness-turn window is documented: Claude Code can start a notification turn the moment any prompt is written, and the two processes share no lock. Round 2 (a9af7b3/bcc7682) BLOCKed on a forgotten agent's resume, a stale fired prompt starting after Stop, and unbounded state. Round 3 (4d0ed9f) BLOCKed on close leaving retryable: true and the cap not applied after Stop or exhaustion. Round 4 (4217083) BLOCKed on records not ordered with their changes, sent keyed only by prompt id, and a late failure during close. Round 5 (25c3094) BLOCKed on a queued retry starting inside Claude Code's own turn and a forgotten agent's later end being lost. Round 6 (cb68b37) BLOCKed on a raw failure during close. Each was fixed with a test (above). Round 7: SHIP at aa4f064. Round 8: SHIP at the head 7694b99a4554bd143c0c3653dbdf3519267910ff, after removing one .expect the crash ratchet flagged.

Reds at the head, all base reds: "cmux-next checks" fails only tree-key-embeds.test.sh on docs/remote-agents-and-cloud.md, which cli/command.rs on base embeds since #25311; this diff embeds nothing. The others (merge tree, same-tree cmux-tui, review link, gate posts, familiar-intent warning) fail the same way on other PRs.

Size: 1539 changed lines over #25264, over the ~800 rule. About 800 of them are tests: 396 integration, 209 in-crate race tests, 87 fake_claude and about 100 unit tests. The code is about 730 lines, one module and its wiring. Review rounds 2 to 6 added the safety checks that made it grow. Splitting the code would land a retry without its skip, start and close checks, which is the very thing the review blocked on.

No capture: daemon-only change. The webview that draws retry/retryable is #25226.

Agent session: big-red
🤖 Generated with Claude Code