[fix]: handle race in `close()` when using API
why
when running evals at high concurrency (~30) with the stagehand API enabled in a region far from the client machine, browser sessions die and the entire eval run hangs indefinitely. the process never exits because v3.close() awaits a promise that never resolves.
the root cause is a race in the shutdown path between:
- the HTTP round-trip of
apiClient.end()(client → API → client), and - the CDP websocket close event arriving from the browser
race sequence:
v3.close()awaitsapiClient.end()(HTTP request/response)- the API requests release immediately, and the browser closes the CDP websocket
- the websocket
"close"event can reach the client before the HTTP response returns - only after the HTTP response returns do we call
ctx.close()→CdpConnection.close()
if the close event wins the race and fires before step (4), then CdpConnection.close() attaches its one-shot
"close" listener too late (the event already happened), so the promise never resolves.
far regions make this more likely because the HTTP RTT is longer, increasing the window where the close event can
beat the response.
two additional issues were found in the same shutdown path:
- inflight CDP calls are never rejected when the websocket closes unexpectedly. pending
send()promises hang forever alongsideclose(), which compounds the original hang - the
_onCdpClosedtransport handler was still registered whenapiClient.end()was called. the server-side websocket close would fire_onCdpClosed->_immediateShutdown()->close({ force: true }), re-entering the close path while the originalclose()was still in progress
this bug only reproduced reliably with regions that are far away from the client machine because:
apiClient.end()awaits a full request/response to the hosted API; higher client-region latency makes that await longer- during that longer window, the server-side websocket close can arrive and be processed before
CdpConnection.close()is called - if the client machine is close to the region, the timing often works out and
close()registers the"close"listener in time
what changed
inpackages/core/lib/v3/understudy/cdp.ts:
- updated
close()to have an early return whenthis.ws.readyState === WebSocket.CLOSEDso that callingclose()on an already-closed websocket resolves immediately instead of hanging. - added
rejectAllInflight(why), which is a private method that iterates theinflightmap and rejects every pending CDP call with aCdpConnectionClosedError, then clears the entry. this is called from both the websocket"close"and"error"handlers in the constructor - updated the
"close"and"error"handlers to callthis.rejectAllInflight(why)beforethis.emitTransportClosed(why)to make sure all pending promises are rejected when the ws dies
in packages/core/lib/v3/v3.ts:
- updated
v3.close()such thatoffTransportClosedruns beforeapiClient.end() - this prevents
_onCdpClosedfrom firing during the API teardown and re-enteringclose({ force: true })
test plan
- added unit tests in
cdp-connection-close.test.tswhich check that:CdpConnection.close()resolves when the websocket is already closed- inflight
send()calls reject when the websocket is terminated
- added an e2e test in
cdp-connection-close.spec.tswhich:- terminates the underlying websocket, then calls
v3.close()and asserts it resolves - terminates the websocket with an inflight CDP call and asserts the pending promise rejects
- terminates the underlying websocket, then calls
- added another e2e test in
cdp-close-api-region.spec.tswhich:- reproduces the full production trigger chain by creating a stagehand instance with
disableAPI: falseandregion: "ap-southeast-1", - calls
v3.close()and asserts it resolves within 30 seconds
- reproduces the full production trigger chain by creating a stagehand instance with
Summary by cubic
Fixes a shutdown race that caused eval runs to hang when using the Stagehand API in non-us-west-2 regions. v3.close() now resolves reliably and inflight CDP requests reject on socket shutdown, addressing Linear STG-1422.
- Bug Fixes
- CdpConnection.close(): return immediately if the WebSocket is already CLOSED.
- Reject inflight CDP calls on "close"/"error" via new CdpConnectionClosedError (exported in public error types).
- Unhook transport-close handler before apiClient.end() to prevent double-close during API teardown.
- Added unit and e2e tests (incl. ap-southeast-1) verifying close resolves and inflight requests reject.