Fix OpenAI cua agent

Merged
TK
8mo ago
+63-12

why

  • OpenAI now requires a screenshot on initial message for computer use agents

what changed

  • added screenshot to initial message
  • imported types from openai sdk for stronger typing

test plan

  • tested locally using operator-example script

Summary by cubic

Send an initial screenshot with the first message in OpenAICUAClient to meet OpenAI’s computer-use requirement and prevent startup errors. Also adopt openai SDK response types for stricter typing.

  • Bug Fixes

    • Capture and attach a high-detail screenshot on the first request when available; gracefully skip on failure.
    • Build the first message using EasyInputMessage, ResponseInputText, and ResponseInputImage; include an optional system message from userProvidedInstructions; make createInitialInputItems async and update method signatures to accept a unified OpenAIRequestInputItem type.
  • Dependencies

    • Add changeset for a patch release of @browserbasehq/stagehand.