PR #25232 - manaflow-ai/cmux

Leo, word for word (2026-10-26): "this needs to retry :("

subsession-card's ask, daemon half (a): "A turn that fails on a transient upstream error (HTTP 503, 529 overloaded, "no non-exhausted ... accounts", "overloaded_error", or a connection reset or EOF to the gateway) retries by itself with backoff, then gives up. It resumes the same session and turn, never a new session. Emit a state on the session or turn that the webview can draw. Non-transient errors (auth, a bad request, a refused permission) never retry."

Why these turns never came back

hub/turn_errors.rs read every one of them as an account limit, so resume_cause returned nothing and the turn sat failed:

is_limit_error matched any 401 in the text. The SR2 gateway's errors name its port, (127.0.0.1:4016) / sr2-gateway…:4016, so every SR2 error read as a sign-in failure. That includes the 503 "no non-exhausted claude accounts" error and a plain ECONNRESET to the gateway.
is_limit_error also matched overloaded, so a 529 overloaded_error and Codex's "Selected model is at capacity. overloaded" went to the fallback-profile path, or to nothing when there was no fallback.

What changed (PR 1 of 2; `_acpmux/retry` is PR 2)

turn_errors: new is_capacity_error (HTTP 503 or 529, overloaded, "at capacity", "no non-exhausted … accounts") and pub fn transient_cause(error) -> Option<&'static str>. It returns capacity, transport, or None for limits, sign-in and anything else. hb-retry-subagent shares this classifier. resume_cause calls it and keeps its own agent_exited branch. Status codes (401, 429, 503, 529) now match only as whole numbers. overloaded is no longer a limit. Transport also covers unexpected eof and early eof.
auto_resume: capacity errors go through the existing continuation path in the same session (prompt id P~resume-n, the harness's own resume), with a continuation text that names the capacity error.
Session row (session/list _meta.acpmux, _acpmux/sessions, _acpmux/session_changed): retry: {attempt, of, reason, nextAt, promptId, turnId}, where reason is capacity | transport | agent_exited and nextAt is epoch ms. It is null when nothing waits, and it clears when the attempt fires, is skipped (Stop, close, a turn that took the chat) or runs out. New protocol types RetrySummary and RetryReason. The IR and the webview's generated types are regenerated.
Events: auto_resume_scheduled gains of, reason and nextAt. auto_resume_exhausted gains reason. Watchers now hear all three auto_resume_* records as _acpmux/session_changed, so the row's retry comes and goes live.
The wait's timer also wakes when its mark goes, so Stop or close records the skip and clears the row at once instead of up to 20 minutes later.

Field names match hb-acpmux's contract (retry-contract.md). subagentId? on the row is hb-retry-subagent's to add.

Decisions

A capacity error does not try the fallback profile first. The SR2 503 comes from the pool that Claude's fallback profile points at, so hopping lands on the same exhausted pool. A 529 is the provider's overload whichever account asks. A hop also costs a fresh agent start (or a transcript rehydrate when the store differs) on what is usually a 15-second blip. Retrying in place at 10 s recovers soonest. Usage limits (429, "hit your limit", quota) and sign-in failures keep going to the fallback, unchanged.
The default delays stay at 10 s, 1 min, 5 min and 20 min, each staggered up to a fifth. Claude Code has already retried internally before it reports the 503, so a shorter first wait would mostly hit the same full pool. When the pool really is out ("next account frees up in 50m"), the later attempts cover it.

Stop conditions (timers)

The one timer is each chat's existing auto-resume wait. It now ends at its delay or when its mark is dropped (Wake's Drop notifies it): Stop, close, a newer turn taking the chat, or the attempts running out. In every case the task then records the skip or fires once and exits. No new observer or loop.

Tests (red first)

Red at 1be746fa: turn_errors::capacity_errors_are_continued_in_place and a_dropped_gateway_connection_is_continued failed (an ECONNRESET naming the gateway's :4016 read as a 401). So did the integration tests auto_resume::capacity::* ("never scheduled") and the reason assertion added to automatic_continuations_stop_after_the_configured_attempts.
Unit: the exact SR2 503 texts (including "retry after 2401s" and "429s" variants), the 529 overloaded_error JSON and "Selected model is at capacity." are capacity. 429 / "hit your limit" / usage limit / 401 / "Not logged in" / invalid API key / 400 invalid_request_error / permission refusals stay None.
Integration (fake agent cap-fail-once: answers with the SR2 503): the turn continues in the same session with the same agent session id and prompt id p-cap~resume-1. While it waits, retry shows in hub.session_summary, in session/list and in _acpmux/sessions. It is null after the attempt fires. Stop clears it at once, and a watcher hears auto_resume_scheduled with retry and then auto_resume_skipped with retry: null. A prompt sent meanwhile clears the row, and the wait is skipped as superseded.
Green at 20c03fae (Blacksmith fast-checks run 40253941821): cargo fmt -p acpmux makes no diff; cargo test -p acpmux --lib passes 813 tests, protocol_ir_is_current among them; --test hub_integration passes 411; clippy --all-targets is clean. The webview's bun run acpmux-protocol:check is current (run 40253470646).

Review

Codex gpt-6-astra (xhigh) review: pending.

No capture: no UI changes here. The webview lane draws retry from these fields.

Compatible with #23921 (restart continuations): it reuses split_attempt, SUFFIX and MAX_ATTEMPTS unchanged, and this PR leaves the doc lines that PR edits untouched.

Agent session: big-red
🤖 Generated with Claude Code