Add `screenshot` option to Extract

Merged
MI
6mo ago
+348-6

why

Some pages do not properly encapsulate the necessary information required to extract the content requested by the user, and can benefit from a hybrid approach with vision.

what changed

  • New extract({ options: { screenshot: true } }) flag. Captures the current viewport as PNG and sends it as an image_url part alongside the a11y tree text in the extraction LLM call.
  • Default is false — existing callers see no behavior change.
  • Gated to AI SDK clients; throws StagehandInvalidArgumentError otherwise (validated in both the handler and inference.extract, with the handler check running before the screenshot is taken).
  • Screenshot capture is bracketed by ensureTimeRemaining() so it respects the extract timeout.
  • Wired end-to-end: public ExtractOptions type, ExtractOptionsSchema, OpenAPI ExtractOptions, StagehandAPIClient.extract wire payload, V3.extract → ExtractHandler → inference.extract → buildExtract{System,User}Prompt.

test plan

  • extract-screenshot.test.ts (new) — multimodal message shape on happy path; rejection for non-AI SDK clients.
  • timeout-handlers.test.ts — handler captures viewport when enabled, skips capture by default, rejects screenshot + non-AI SDK client without touching page.screenshot or the snapshot pipeline.
  • api-client-serialization.test.ts (renamed from api-client-observe-variables.test.ts) — wire payload preserves options.screenshot.

Summary by cubic

Adds a screenshot option to extract() that sends the current viewport screenshot with the accessibility tree to improve results on visually-driven pages.

  • New Features
    • extract({ options: { screenshot: boolean } }) captures a PNG of the current viewport and includes it with the accessibility tree; defaults to false; only supported with aisdk clients (throws StagehandInvalidArgumentError otherwise).
    • Prompts are consolidated and multimodal when enabled: system and user prompts mention the screenshot and using it with the accessibility tree; attaches an image_url data URI; logs when a screenshot is used.
    • Public API, types, and OpenAPI updated (ExtractOptions.screenshot); the API client forwards the flag; tests cover prompt shape, viewport capture, default-off behavior, timeout guards, client-type validation, and serialization.