Default `AnthropicModel`'s `max_tokens` to the model's maximum output, streaming such requests behind the scenes

Merged
DO
1mo ago
+53748-45577

Follows #9025, which raised the default to 16384; see the discussion on #2553 and #7468.

Why we make these changes

AnthropicModel has to send max_tokens. When the user doesn't set one, it now sends the model's maximum output, so responses are only cut off at the model's own limit. That's how the output limit behaves on APIs where it's optional, like OpenAI's and Google's. Thinking counts toward max_tokens, and current Claude models think by default, so a fixed default can still run out before the answer.

For that many tokens the Anthropic SDK refuses a non-streaming request, so run() now streams the request behind the scenes. Its response is built by AnthropicStreamedResponse, the same code as run_stream() and run_stream_events(), so there's one streaming path. The request is sent once; the previous fallback rebuilt it, mapping messages and downloading files twice.

[!WARNING]
Compatibility impact: Requests without max_tokens to Claude Sonnet 4.5 and later now send 64,000 or 128,000 and are streamed behind the scenes. On Bedrock (AsyncAnthropicBedrock) that reserves more TPM quota up front. A timeout now limits the wait between streamed chunks rather than the whole response. Tool call args in those run() results are the streamed JSON string rather than a dict, as on streamed runs; args_as_dict() gives the same value, and args was always typed str | dict[str, Any] | None.

Why this can ship in a minor release: it removes an artificial output cap. The responses run() builds carry the same content: the new parity test checks every recorded streamed response against the complete message.

Migration: Set max_tokens to keep a lower limit and non-streaming requests.

New public surface

  • AnthropicModelProfile.anthropic_max_output_tokens: the per-model maximum from Anthropic's Models API. It's 128K for Opus 4.6+, Sonnet 4.6+, the 5.x models, Fable and Mythos 5, and 64K for Haiku, Sonnet and Opus 4.5. Unknown models get 16384, and models older than Sonnet 4.5 keep 4096 (#9025).

Streaming fixes, for run() and streamed runs alike

  • A server tool call made from code execution (e.g. web_search with dynamic filtering) keeps its args, which arrive on the block start rather than as deltas. They were None.
  • An error event inside a stream arrives with HTTP status 200. It now maps to its real status by error type, so it raises ModelHTTPError (e.g. 529 for overloaded_error) instead of a bare ModelAPIError. The stream's first event is read within the request's retries, so an expired code-execution container is retried when streaming too, on Anthropic's current 404 not_found_error as well as the old 500.
  • The 1-hour cache-write count is capped at cache_creation_input_tokens after merging streamed usage. A compaction stream otherwise keeps the start event's stale split, which double-counts 1-hour writes and prices the request at None.
  • With a custom timeout the SDK doesn't require streaming, so a default max_tokens above its non-streaming limit is streamed by AnthropicModel itself.

Verification

  • tests/models/anthropic/test_stream_parity.py replays all 312 recorded streamed Anthropic responses through AnthropicStreamedResponse and through the SDK's accumulated BetaMessage, and asserts identical ModelResponses, comparing tool call args by value. It found the dropped server tool call args above.
  • tests/models/anthropic/test_max_tokens.py: VCR cases for the model max, adaptive and extended thinking, a custom timeout, and an explicit max_tokens, asserting the wire max_tokens and stream. test_implicit_stream.py covers a stream that breaks before its first event or mid-response (ModelAPIError) and an empty one.
  • About 190 cassettes whose requests now stream were re-recorded live. The diffs are fresh content (ids, token counts, wording); part types and fields are unchanged. The re-recording also exposed unrelated drift: robots.txt-blocked URLs, the retired gemini-2.0-flash, DeepWiki's renamed MCP tool, and tool-search tests reading the old VCR.py format.
  • clai2 no longer sets max_tokens for classic thinking, since the default now leaves room for the answer.

What changes for existing users

Claude Sonnet 4.5 and later can produce responses up to the model's maximum without max_tokens, and run() streams those requests. Their tool call args are JSON strings, as on streamed runs.

Checklist

  • No breaking changes in accordance with the version policy.
  • Any permitted compatibility impact has a label, warning, migration, release note, and exact API-check waiver. (No public API is removed or changed, so no API-check waiver applies.)
  • PR title is fit for the release changelog.

šŸ¤– Generated with Claude Code