fix(core): parse screenshot MIME in GoogleCUAClient function responses
Closed
YA
4mo ago+83+83+01234567890123456789-5-5-0123456789
why
Closes #2046. GoogleCUAClient builds function-response image parts with inlineData.mimeType = "image/png" and strips the data URL prefix using a PNG-only regex. If a screenshot is JPEG/WebP (or anything non-PNG), the metadata is wrong and the regex no-ops — the whole data URL ends up as the payload.
what changed
- Added
parseImageDataUrl(input)helper that extracts{ mimeType, data }from animage/<type>data URL, falling back toimage/png+ the raw input for unrecognized formats (preserves prior behavior). - Replaced the hardcoded PNG strip + hardcoded
mimeTypeat the function-response call site with the helper's parsed values. - No change to
captureScreenshot()— it still normalizes raw provider input to PNG data URLs, matching the issue's stated current contract.
test plan
New packages/core/tests/unit/google-cua-client.test.ts covering:
- PNG data URL →
image/png+ base64 payload - JPEG data URL →
image/jpeg+ base64 payload - WebP data URL →
image/webp+ base64 payload - Raw base64 → fallback to
image/pngwith the input as data - Non-image data URL (e.g.
text/plain) → fallback toimage/png
Also ran pnpm run typecheck and pnpm run eslint in packages/core — both clean.
Summary by cubic
Fixes screenshot MIME parsing in GoogleCUAClient function responses by deriving the type and base64 payload from image data URLs, including newline-wrapped payloads. Addresses Linear issue #2046 and ensures JPEG/WebP screenshots set the correct inlineData.mimeType, with a PNG fallback for raw inputs.
- Bug Fixes
- Added
parseImageDataUrl(input)to extract{ mimeType, data }fromdata:image/*;base64,..., handling newline-wrapped base64. - Replaced PNG-only strip and hardcoded
mimeTypewith parsed values. - Added unit tests for PNG, JPEG, WebP, newline-wrapped data, raw base64, and non-image data URLs.
- Added