#2300 fix(core): declare CUA screenshot media type at capture boundary

why

Closes #2046. This is the reshaped version of #2159, following the approach @seanmcguire12 outlined when closing that PR.

setScreenshotProvider returned a bare base64 string, so every CUA client had to independently infer or hardcode the media type — all four assumed image/png. A non-PNG screenshot (e.g. a JPEG from a custom provider) was then mislabeled as PNG in the provider function-response payload, which is the root of #2046. Clients also stripped a hardcoded data:image/png;base64, prefix by regex, so any other prefix silently broke.

what changed

Move the media-type declaration to the capture boundary. setScreenshotProvider now returns an explicit payload:

export interface ScreenshotProviderResult {
  base64: string;
  mediaType: "image/png" | "image/jpeg";
}

• Default handler (v3CuaAgentHandler) captures PNG explicitly (type: "png") and returns { base64, mediaType: "image/png" }, so the default is unchanged.
• Anthropic: media_type: screenshot.mediaType, data: screenshot.base64 (drops the .replace(/^data:image\/png;base64,/, "")).
• Google: mimeType: screenshot.mediaType (drops the PNG-only prefix strip).
• OpenAI / Microsoft: build data:${screenshot.mediaType};base64,${screenshot.base64}.
• options.base64Image (caller-supplied) still defaults to image/png, preserving existing behavior.

ScreenshotProviderResult is exported from the public entrypoint.

testing

• New cua-screenshot-mediatype.test.ts: asserts a non-PNG (image/jpeg) media type is honored by all four clients' captureScreenshot(), and that the options.base64Image path still defaults to png.
• Updated the public API type test for setScreenshotProvider(...) and the Anthropic/Microsoft CUA client tests to the new provider shape.
• pnpm --filter @browserbasehq/stagehand run typecheck passes; the CUA + public-API unit suites are green (55 tests).

---

Summary by cubic

Declare the screenshot media type at the capture boundary and pass it through all CUA clients. Fixes non‑PNG screenshots being mislabeled as PNG and removes PNG-only prefix stripping.

• Bug Fixes

  • setScreenshotProvider now returns { base64, mediaType } (ScreenshotProviderResult) instead of a string.
  • Default handler explicitly captures PNG and returns image/png.
  • Clients: Anthropic/Google pass mediaType through; OpenAI/Microsoft build data:${mediaType};base64,${base64}; removed PNG-only prefix regex.
  • options.base64Image still defaults to image/png.
  • Added tests validating JPEG flows through all clients; updated public API type tests.

• Migration

  • If you provide a custom setScreenshotProvider, return { base64, mediaType: "image/png" | "image/jpeg" } instead of a base64 string.
  • No changes needed if you use the built-in handler.