fix(core): infer Anthropic tool_result image MIME type
Closed
BA
4mo ago+133+133+012345678901234567890123456789-4-4-0123456789
why
Anthropic CUA computer tool results currently hardcode media_type: "image/png" and strip only a PNG prefix from screenshot data URLs.
That is brittle if screenshot input is non-PNG (for example JPEG/WebP), and can produce mismatched metadata or malformed image payloads.
Fixes #2035.
what changed
- Added a small internal parser in
AnthropicCUAClientto extract:media_typefrom image data URLs- base64 payload without the data URL prefix
- Applied parsed MIME/data in both computer-tool image paths:
- normal success tool result
- error fallback tool result
- Kept a PNG fallback path for malformed/unexpected screenshot strings.
- Added focused unit tests in
packages/core/tests/unit/anthropic-cua-client.test.tsfor:- JPEG MIME parsing in success path
- PNG fallback on non-data URL input
- MIME/data handling in error path retry screenshot payload
test plan
npm.cmd exec prettier -- --check packages/core/lib/v3/agent/AnthropicCUAClient.ts packages/core/tests/unit/anthropic-cua-client.test.tsnode node_modules/vitest/vitest.mjs run --config .tmp-vitest-unit-config.mjsfrompackages/core(temporary local config targetingtests/unit/anthropic-cua-client.test.ts)
Summary by cubic
Infer MIME type and base64 data from screenshot data URLs for Anthropic CUA tool_result images instead of forcing PNG. Prevents mismatched metadata and malformed image payloads. Fixes #2035.
- Bug Fixes
- Added a parser in
AnthropicCUAClientto extractmedia_typeand base64 data from image data URLs. - Used parsed values in both success and error
computertool_result paths, with a PNG fallback for malformed inputs. - Added unit tests for JPEG parsing, fallback behavior, and error-path screenshot payloads.
- Added a parser in