fix(session): recover from stalled model requests

AN
8d ago
+1217-14

Issue for this PR

Closes #1424

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Why. A session never recovers when a model request goes quiet. In a recorded benchmark, 10 of 135 sessions on one model (GPT-6.1 Sol via Bedrock Mantle) logged a request and then nothing until the harness killed them at its 30 minute limit: no error, no retry, and in 7 of the 10 the finish-time validators never ran. (10 of 135: estimate from recorded data.)

Root cause (verified in code).

  • No first-byte timeout was armed for most providers (only OpenAI and Altimate Base set one), so a request the server never answered waited forever.
  • The existing 5 minute idle watchdog fired, but its error became an unknown error that was not retried, so the session ended.
  • Bedrock Converse event streams are binary, not SSE, and bypassed the idle watchdog.
  • A retry after a mid-stream failure left the failed attempt's text and reasoning in the message, and re-requesting after a dispatched tool call could re-run the tool.

What changed (before, after).

  • Never answered: hung, now aborted at 300s and retried. Silent mid-stream: ended as an error, now retried. Reset mid-stream: retried but partial output stayed, now removed first.
  • Upstream OpenCode already maps both timeouts to retryable errors and defaults the header timeout to 300s (dev #46903). This aligns with that (here only the idle watchdog's stream error is made retryable) and adds the partial-output rule and the Converse coverage. Defaults: first byte 300s (new), idle 300s (unchanged). The slowest healthy generation in 6,586 recorded steps was 51s.
  • Partial-output rule: before a retry, text, reasoning, step-start and still-pending tool parts written by the failed attempt are removed. If a tool's execute() began (counted at the tool wrapper) or a step finished, it is not retried (a tool is never re-run) and the error says why.
  • After 5 retries the session ends with "The model stopped responding ... gave up after 5 retries". A user cancel is never retried and still ends the wait at once.
  • Config: provider options headerTimeout and chunkTimeout (ms; headerTimeout false disables the check), see providers.md.

Spec. A stalled request is aborted and retried with the existing backoff at most 5 times; partial output is never duplicated; a dispatched tool is never re-run; exhaustion ends with a clear error; cancel is immediate.

Tenant/user impact. A request silent for 300s is now retried instead of hanging. Users could also notice: (1) a persistent stall takes about 31 minutes to surface at the default (6 attempts of 300s plus backoff), so lower headerTimeout where that matters; (2) the 300s first-byte deadline now also bounds custom fetches such as Vertex credential acquisition; a healthy request silent for over 300s before headers (slow local prompt evaluation, non-streaming calls) would be cut off, so raise headerTimeout there; (3) the no-retry-after-started-tool rule covers every retryable error: a mid-stream 5xx after a tool started used to be retried (re-running the tool) and now ends with an error, and, as for any errored step, its tool history is not replayed next turn; (4) the TUI shows "Model stopped responding" for the exhausted error; (5) a Stop just before a backoff ends the wait at once. Validators are unchanged: they run after a clean stop, so a recovered stall reaches them; an exhausted one skips them.

Deployment readiness. No migration, dependency or config; revert restores old behaviour.

How did you verify your code works?

  • Unit: error classification, abort stays an abort, discard rule, tool-call ids.
  • Integration with real components: the real session processor, LLM stream, provider fetch wrapper and AI SDK against a local raw TCP server speaking the OpenAI-compatible streaming protocol: never answered, silent mid-stream, Converse content type, socket reset, partial, dispatched and in-flight tool calls (tool runs once), bounded retries (6 requests), cancel during the wait and the backoff, a slow healthy stream not cut off, all timers cleared, the 300s default armed. 26 tests in the new file pass; on main 13 of the first 19 fail (the rest are guards and tests of the new module).
  • Reproduction with defaults: on main a silent server left the session pending for 170s until stopped; with the change it recovered after 300s.
  • Also run: typecheck, test/session, test/provider, TUI, chunk-timeout tests; marker check.
  • Live provider and Bedrock SDK frame decoding: Not run (no model spend). Recovery was verified against a fake server reproducing the stall shapes, not the real provider; real gateways may stall differently.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

šŸ¤– Generated with Claude Code

[!NOTE]
Medium Risk
Changes core provider fetch timeouts, session retry semantics, and tool-call safety rules; mis-tuned headerTimeout could cut off legitimately slow streams, and the no-retry-after-tool-start rule alters behavior for any retryable error mid-step.

Overview
Sessions that hung or died on silent providers now abort stalled requests, retry with backoff (up to 5 times), and clean up partial output so retries do not duplicate streamed text or re-run tools.

Provider layer: Most providers get a 300s default headerTimeout (configurable via options.headerTimeout, or false to disable). The stream idle watchdog now covers Amazon Bedrock event streams as well as SSE. Custom fetches that ignore abort signals (e.g. Vertex credential work) are raced against the first-byte deadline.

Session layer: Header and idle-stream timeouts map to retryable "The model stopped responding" errors. Before each retry, StallRecovery removes text, reasoning, step-start, and pending tool parts from the failed attempt; retries are blocked once a tool has started executing or a step has finished, with a clear error message. User cancel skips retry and fails fast during retry backoff if the signal is already aborted.

Docs / TUI: providers.md documents headerTimeout and chunkTimeout; notifications treat the new exhausted-stall message as "Model stopped responding".

Reviewed by Cursor Bugbot for commit a689e5ad8cc518906cc1e27be7ab8367005840e0. Bugbot is set up for automated code reviews on this repo. Configure here.