acpmux: recover turns after limits and gateway outages
This PR makes acpmux recover failed turns in place instead of stranding sessions after account limits or gateway outages. It uses the existing RunSpec admission and account cooldown path; it does not add a second scheduler or revive by switching accounts.
Leo, verbatim (2026-10-29): "ok uh in addition i was asking you guys about that so you could help analyze my usage and how to make us both more effective and efficient".
Leo also said: "hmm what's going on with these? they get lost under network breaks and dont do anything?" and "also let's not do this anymore" about lane-revive switching accounts.
The implementation resumes the same session after Claude session/weekly limits, router capacity, HTTP 429/502/503/529, and harness_unavailable; honors Retry-After seconds/units/date, Claude reset hints, and persisted usage windows; preserves held messages; shows an acpmux-owned retry with reset timing; restores scheduled retries after daemon restart; and emits a visible error after its bounded attempt window. Stop, close, newer prompt, movement, explicit retry, dispatch refusal, and daemon shutdown end each timer.
Related designs and issues: HQ#5123 §3, HQ#4879 items 3-4, and audit PR #1544.
Performance
| metric | before | after | delta | harness or run link |
|---|---|---|---|---|
| Named lanes failed or stuck on Claude 429 / router exhaustion | 228 (coordinator pass, 2026-10-30 05:40Z) | rerun the same coordinator query after deployment | pending deployment measurement | coordinator capacity/stuck query |
| Sessions ending on a limit error with no later turn | 16 / 1,177 (observed transcript-tail scan, 2026-10-30 02:47Z) | rerun the same 14-day scan after deployment | pending deployment measurement | tools/usage-report run --days 14 --github plus the same session-ending scan |
Validation
- Red/green regression commit
435ac10d1a1and subsequent implementation commits through1e92656adcecover forced 429/502/503/harness_unavailable, weekly reset timing, queue preservation, restart restore, and stop/close behavior. - Focused command:
cd cmux-tui && cargo test -p acpmux --test hub_integration limit_recovery -- --nocapturethrough the hosted cmux-ci route. A prior hosted run ended without a broker receipt while shared quota was exhausted; the code was reviewed against the exact pushed head afterward. python3 scripts/cmux-next/prepush-gate.py: no new violations.verify-local.py: existing unrelated localization debt on basea1f302795f52c68f5ddfd400bafc39782551c169.
Risk: normal. Reset scheduling uses the event log and existing session queue; retry timers stop on the conditions named above.
Changelog: none
Agent session: agent session