Add `screenshot` option to Extract
Merged
MI
6mo ago+348+348+012345678901234567890123456789-6-6-0123456789
why
Some pages do not properly encapsulate the necessary information required to extract the content requested by the user, and can benefit from a hybrid approach with vision.
what changed
- New
extract({ options: { screenshot: true } })flag. Captures the current viewport as PNG and sends it as animage_urlpart alongside the a11y tree text in the extraction LLM call. - Default is
false— existing callers see no behavior change. - Gated to AI SDK clients; throws
StagehandInvalidArgumentErrorotherwise (validated in both the handler andinference.extract, with the handler check running before the screenshot is taken). - Screenshot capture is bracketed by
ensureTimeRemaining()so it respects the extract timeout. - Wired end-to-end: public
ExtractOptionstype,ExtractOptionsSchema, OpenAPIExtractOptions,StagehandAPIClient.extractwire payload,V3.extract→ExtractHandler→inference.extract→buildExtract{System,User}Prompt.
test plan
extract-screenshot.test.ts(new) — multimodal message shape on happy path; rejection for non-AI SDK clients.timeout-handlers.test.ts— handler captures viewport when enabled, skips capture by default, rejects screenshot + non-AI SDK client without touchingpage.screenshotor the snapshot pipeline.api-client-serialization.test.ts(renamed fromapi-client-observe-variables.test.ts) — wire payload preservesoptions.screenshot.
Summary by cubic
Adds a screenshot option to extract() that sends the current viewport screenshot with the accessibility tree to improve results on visually-driven pages.
- New Features
extract({ options: { screenshot: boolean } })captures a PNG of the current viewport and includes it with the accessibility tree; defaults to false; only supported withaisdkclients (throwsStagehandInvalidArgumentErrorotherwise).- Prompts are consolidated and multimodal when enabled: system and user prompts mention the screenshot and using it with the accessibility tree; attaches an
image_urldata URI; logs when a screenshot is used. - Public API, types, and OpenAPI updated (
ExtractOptions.screenshot); the API client forwards the flag; tests cover prompt shape, viewport capture, default-off behavior, timeout guards, client-type validation, and serialization.