thanks @yawbtng for the contribution here!

why

Closes #2046. This is the reshaped version of #2159, following the approach @seanmcguire12 outlined when closing that PR.

setScreenshotProvider returned a bare base64 string, so every CUA client had to independently infer or hardcode the media type — all four assumed image/png. A non-PNG screenshot (e.g. a JPEG from a custom provider) was then mislabeled as PNG in the provider function-response payload, which is the root of #2046. Clients also stripped a hardcoded data:image/png;base64, prefix by regex, so any other prefix silently broke.

what changed

Move the media-type declaration to the capture boundary. setScreenshotProvider now returns an explicit payload:

export interface ScreenshotProviderResult {
  base64: string;
  mediaType: "image/png" | "image/jpeg";
}

• Default handler (v3CuaAgentHandler) captures PNG explicitly (type: "png") and returns { base64, mediaType: "image/png" }, so the default is unchanged.
• Anthropic: media_type: screenshot.mediaType, data: screenshot.base64 (drops the .replace(/^data:image\/png;base64,/, "")).
• Google: mimeType: screenshot.mediaType (drops the PNG-only prefix strip).
• OpenAI / Microsoft: build data:${screenshot.mediaType};base64,${screenshot.base64}.
• options.base64Image (caller-supplied) still defaults to image/png, preserving existing behavior.

ScreenshotProviderResult is exported from the public entrypoint.

testing

• New cua-screenshot-mediatype.test.ts: asserts a non-PNG (image/jpeg) media type is honored by all four clients' captureScreenshot(), and that the options.base64Image path still defaults to png.
• Updated the public API type test for setScreenshotProvider(...) and the Anthropic/Microsoft CUA client tests to the new provider shape.
• pnpm --filter @browserbasehq/stagehand run typecheck passes; the CUA

• public-API unit suites are green (55 tests).

---

Summary by cubic

Declare the screenshot media type at the capture boundary and thread it through all CUA clients to fix mislabeled images and remove PNG-only prefix handling. Non‑PNG screenshots (e.g. JPEG) now work end-to-end.

• Bug Fixes

  • setScreenshotProvider now returns ScreenshotProviderResult ({ base64, mediaType }) instead of a string.
  • Default handler captures PNG (type: "png") and returns image/png.
  • Clients: Anthropic/Google pass mediaType through; OpenAI/Microsoft build data:${mediaType};base64,${base64}; removed PNG-only prefix stripping.
  • captureScreenshot({ base64Image }) accepts optional mediaType; defaults to image/png.
  • Added tests covering JPEG across clients and updated public API type tests.

• Migration

  • If you use a custom setScreenshotProvider, return { base64, mediaType: "image/png" | "image/jpeg" } instead of a base64 string. No changes needed with the built-in handler.

why

what changed

test plan