CodeRouter: hold capacity errors on the same model instead of failing fast

Merged
TE
1mo ago
+782-35

A capacity blip ended autonomous agent turns. When every Claude or Codex account was cooling, or four accounts had failed, CodeRouter answered 503 overloaded_error. Codex and Claude Code show that as a final error and stop until a human types "continue" (#15301).

CodeRouter now holds the request and retries the same model instead of failing fast:

  • When a routing round ends on a transient failure (429, 5xx/529, an overloaded or "model at capacity" SSE event before any output, a transport error), the proxy waits and replays the same body. Waits use jittered exponential backoff (0.5 s → 15–30 s) and never end before the soonest account cooldown, which already carries the upstream retry-after.
  • The hold lasts up to CODEROUTER_CAPACITY_HOLD_MS (default 20 minutes). It is capped by the existing 25-minute header budget, so the function still answers before its 30-minute maxDuration.
  • It never holds once response bytes reached the client, and it never substitutes a model.
  • It fails at once, as today, when no account recovers within the budget: every account has a revoked credential, or a Codex workspace quota resets after the budget. Claude's selector skips invalid_credential cooldowns when computing the recovery time; Codex gets a small nextCapacityAvailableAt query for the same purpose.
  • route_events gains held_ms and hold_count, and the PostHog trace carries coderouter_held_ms / coderouter_hold_count, so capacity pain shows up as latency.

Deploy order: ClickHouse migration web/db/clickhouse/006_route_events_capacity_hold.sql (additive ADD COLUMN IF NOT EXISTS ... DEFAULT 0) must be applied to coderouter_dev and coderouter before merge. Otherwise route-event inserts carry unknown columns.

Part 2 of #15301 (agents auto-resume after a retryable failure) is a separate PR.

Validation

  • Commit 1 adds the tests only; commit 2 adds the change.
  • bun test tests/coderouter-*.test.ts*: 506 pass, 0 fail (rebased on main). CI: commit 1 red (web typecheck and web tests), commit 2 green (59/59). New tests: a 529 → 429 → transport → 529 → 200 storm on one Claude account returns 200 with four holds and the same model on every replay; the same for Codex, with each wait honoring the recorded cooldown; the budget cap; no hold for a revoked credential or an hour-long quota; no replay after output started.
  • tsc --noEmit clean; lint:complexity clean.
  • Production build against a mocked upstream (local next build + next start; this machine cannot reach cmux's Vercel previews). A throwaway, unpushed branch swaps in stub auth, one stub account with in-memory cooldowns, and a mock upstream, then drives the real /v1/messages and /v1/responses routes. Every storm shorter than the budget reached the client as a 200 on the same model; the same build with CODEROUTER_CAPACITY_HOLD_MS=0 returns the 529 in 2.3 s, which is the old behavior.
Transcript 
$ # Claude: 3 x 529, then 200 (hold on, default 20 min budget)
=== claude-529 (3 x 529) after 94s
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"held and served"}}
HTTP 200 in 93.024501s

$ # Claude: 2 x 429 retry-after 5, then 200 (same account, concurrent)
=== claude-429 (2 x 429) after 68s
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"held and served"}}
HTTP 200 in 67.095940s

$ # Codex: 2 x "Selected model is at capacity" SSE event, then 200
data: {"type":"response.output_text.delta","delta":"held and served"}
HTTP 200 in 121.558265s
elapsed 122s

# server log (mock upstream)
[mock-upstream +   0.0s] claude-529 client request received (failures=3, kind=529)
[mock-upstream +   1.7s] claude-529 upstream POST model=claude-sonnet-4-5
[mock-upstream +   1.7s] claude-529 attempt 1 -> 529
[mock-upstream +   1.7s] claude-529 cooldown verify-claude-1 20000 ms (upstream_unavailable)
[mock-upstream +   0.0s] claude-429 client request received (failures=2, kind=429)
[mock-upstream +  20.7s] claude-429 upstream POST model=claude-sonnet-4-5
[mock-upstream +  20.7s] claude-429 attempt 1 -> 429
[mock-upstream +  20.7s] claude-429 cooldown verify-claude-1 5000 ms (rate_limited)
[mock-upstream +  25.9s] claude-429 upstream POST model=claude-sonnet-4-5
[mock-upstream +  25.9s] claude-429 attempt 2 -> 429
[mock-upstream +  25.9s] claude-429 cooldown verify-claude-1 5000 ms (rate_limited)
[mock-upstream +  36.1s] claude-529 upstream POST model=claude-sonnet-4-5
[mock-upstream +  36.1s] claude-529 attempt 2 -> 529
[mock-upstream +  36.1s] claude-529 cooldown verify-claude-1 20000 ms (upstream_unavailable)
[mock-upstream +  61.6s] claude-429 upstream POST model=claude-sonnet-4-5
[mock-upstream +  61.6s] claude-429 attempt 3 -> 200 stream
[mock-upstream +  61.8s] claude-429 client response status=200
[mock-upstream +  65.5s] claude-529 upstream POST model=claude-sonnet-4-5
[mock-upstream +  65.5s] claude-529 attempt 3 -> 529
[mock-upstream +  65.5s] claude-529 cooldown verify-claude-1 20000 ms (upstream_unavailable)
[mock-upstream +  89.0s] claude-529 upstream POST model=claude-sonnet-4-5
[mock-upstream +  89.0s] claude-529 attempt 4 -> 200 stream
[mock-upstream +  89.0s] claude-529 client response status=200
[mock-upstream +   0.0s] codex-capacity client request received (failures=2, kind=529)
[mock-upstream +   0.0s] codex-capacity upstream POST model=gpt-5.6-sol
[mock-upstream +   0.0s] codex-capacity attempt 1 -> capacity SSE event
[mock-upstream +   0.2s] codex-capacity cooldown verify-codex-1 60000 ms (model_capacity)
[mock-upstream +  60.5s] codex-capacity upstream POST model=gpt-5.6-sol
[mock-upstream +  60.5s] codex-capacity attempt 2 -> capacity SSE event
[mock-upstream +  60.5s] codex-capacity cooldown verify-codex-1 60000 ms (model_capacity)
[mock-upstream + 121.1s] codex-capacity upstream POST model=gpt-5.6-sol
[mock-upstream + 121.1s] codex-capacity attempt 3 -> 200 stream
[mock-upstream + 121.1s] codex-capacity client response status=200

$ # Same build, CODEROUTER_CAPACITY_HOLD_MS=0 (old fail-fast), warm server, 1 x 529
run 2 HTTP 529 in 2.254531s
[mock-upstream +   0.0s] nohold-2 client request received (failures=1, kind=529)
[mock-upstream +   0.0s] nohold-2 upstream POST model=claude-sonnet-4-5
[mock-upstream +   0.0s] nohold-2 attempt 1 -> 529
[mock-upstream +   0.5s] nohold-2 cooldown verify-claude-1 20000 ms (upstream_unavailable)
[mock-upstream +   0.7s] nohold-2 client response status=529

🤖 Generated with Claude Code