fix(core): infer Anthropic tool_result image MIME type

Closed
BA
4mo ago
+133-4

why

Anthropic CUA computer tool results currently hardcode media_type: "image/png" and strip only a PNG prefix from screenshot data URLs.

That is brittle if screenshot input is non-PNG (for example JPEG/WebP), and can produce mismatched metadata or malformed image payloads.

Fixes #2035.

what changed

  • Added a small internal parser in AnthropicCUAClient to extract:
    • media_type from image data URLs
    • base64 payload without the data URL prefix
  • Applied parsed MIME/data in both computer-tool image paths:
    • normal success tool result
    • error fallback tool result
  • Kept a PNG fallback path for malformed/unexpected screenshot strings.
  • Added focused unit tests in packages/core/tests/unit/anthropic-cua-client.test.ts for:
    • JPEG MIME parsing in success path
    • PNG fallback on non-data URL input
    • MIME/data handling in error path retry screenshot payload

test plan

  • npm.cmd exec prettier -- --check packages/core/lib/v3/agent/AnthropicCUAClient.ts packages/core/tests/unit/anthropic-cua-client.test.ts
  • node node_modules/vitest/vitest.mjs run --config .tmp-vitest-unit-config.mjs from packages/core (temporary local config targeting tests/unit/anthropic-cua-client.test.ts)

Summary by cubic

Infer MIME type and base64 data from screenshot data URLs for Anthropic CUA tool_result images instead of forcing PNG. Prevents mismatched metadata and malformed image payloads. Fixes #2035.

  • Bug Fixes
    • Added a parser in AnthropicCUAClient to extract media_type and base64 data from image data URLs.
    • Used parsed values in both success and error computer tool_result paths, with a PNG fallback for malformed inputs.
    • Added unit tests for JPEG parsing, fallback behavior, and error-path screenshot payloads.