Fix OpenAI cua agent
Merged
TK
8mo ago+63+63+01234567890123456789-12-12-01234567890123456789
why
- OpenAI now requires a screenshot on initial message for computer use agents
what changed
- added screenshot to initial message
- imported types from openai sdk for stronger typing
test plan
- tested locally using operator-example script
Summary by cubic
Send an initial screenshot with the first message in OpenAICUAClient to meet OpenAI’s computer-use requirement and prevent startup errors. Also adopt openai SDK response types for stricter typing.
-
Bug Fixes
- Capture and attach a high-detail screenshot on the first request when available; gracefully skip on failure.
- Build the first message using
EasyInputMessage,ResponseInputText, andResponseInputImage; include an optional system message fromuserProvidedInstructions; makecreateInitialInputItemsasync and update method signatures to accept a unifiedOpenAIRequestInputItemtype.
-
Dependencies
- Add changeset for a patch release of
@browserbasehq/stagehand.
- Add changeset for a patch release of