fix(core): parse screenshot MIME in GoogleCUAClient function responses

Closed
YA
4mo ago
+83-5

why

Closes #2046. GoogleCUAClient builds function-response image parts with inlineData.mimeType = "image/png" and strips the data URL prefix using a PNG-only regex. If a screenshot is JPEG/WebP (or anything non-PNG), the metadata is wrong and the regex no-ops — the whole data URL ends up as the payload.

what changed

  • Added parseImageDataUrl(input) helper that extracts { mimeType, data } from an image/<type> data URL, falling back to image/png + the raw input for unrecognized formats (preserves prior behavior).
  • Replaced the hardcoded PNG strip + hardcoded mimeType at the function-response call site with the helper's parsed values.
  • No change to captureScreenshot() — it still normalizes raw provider input to PNG data URLs, matching the issue's stated current contract.

test plan

New packages/core/tests/unit/google-cua-client.test.ts covering:

  • PNG data URL → image/png + base64 payload
  • JPEG data URL → image/jpeg + base64 payload
  • WebP data URL → image/webp + base64 payload
  • Raw base64 → fallback to image/png with the input as data
  • Non-image data URL (e.g. text/plain) → fallback to image/png

Also ran pnpm run typecheck and pnpm run eslint in packages/core — both clean.


Summary by cubic

Fixes screenshot MIME parsing in GoogleCUAClient function responses by deriving the type and base64 payload from image data URLs, including newline-wrapped payloads. Addresses Linear issue #2046 and ensures JPEG/WebP screenshots set the correct inlineData.mimeType, with a PNG fallback for raw inputs.

  • Bug Fixes
    • Added parseImageDataUrl(input) to extract { mimeType, data } from data:image/*;base64,..., handling newline-wrapped base64.
    • Replaced PNG-only strip and hardcoded mimeType with parsed values.
    • Added unit tests for PNG, JPEG, WebP, newline-wrapped data, raw base64, and non-image data URLs.