[fix]: handle race in `close()` when using API

Merged
SE
9mo ago
+302-9

why

when running evals at high concurrency (~30) with the stagehand API enabled in a region far from the client machine, browser sessions die and the entire eval run hangs indefinitely. the process never exits because v3.close() awaits a promise that never resolves.

the root cause is a race in the shutdown path between:

  • the HTTP round-trip of apiClient.end() (client → API → client), and
  • the CDP websocket close event arriving from the browser

race sequence:

  1. v3.close() awaits apiClient.end() (HTTP request/response)
  2. the API requests release immediately, and the browser closes the CDP websocket
  3. the websocket "close" event can reach the client before the HTTP response returns
  4. only after the HTTP response returns do we call ctx.close() → CdpConnection.close()

if the close event wins the race and fires before step (4), then CdpConnection.close() attaches its one-shot
"close" listener too late (the event already happened), so the promise never resolves.

far regions make this more likely because the HTTP RTT is longer, increasing the window where the close event can
beat the response.

two additional issues were found in the same shutdown path:

  • inflight CDP calls are never rejected when the websocket closes unexpectedly. pending send() promises hang forever alongside close(), which compounds the original hang
  • the _onCdpClosed transport handler was still registered when apiClient.end() was called. the server-side websocket close would fire _onCdpClosed -> _immediateShutdown() -> close({ force: true }), re-entering the close path while the original close() was still in progress

this bug only reproduced reliably with regions that are far away from the client machine because:

  • apiClient.end() awaits a full request/response to the hosted API; higher client-region latency makes that await longer
  • during that longer window, the server-side websocket close can arrive and be processed before CdpConnection.close() is called
  • if the client machine is close to the region, the timing often works out and close() registers the "close" listener in time

what changed

inpackages/core/lib/v3/understudy/cdp.ts:

  • updated close() to have an early return when this.ws.readyState === WebSocket.CLOSED so that calling close() on an already-closed websocket resolves immediately instead of hanging.
  • added rejectAllInflight(why), which is a private method that iterates the inflight map and rejects every pending CDP call with a CdpConnectionClosedError, then clears the entry. this is called from both the websocket "close" and "error" handlers in the constructor
  • updated the "close" and "error" handlers to call this.rejectAllInflight(why) before this.emitTransportClosed(why) to make sure all pending promises are rejected when the ws dies

in packages/core/lib/v3/v3.ts:

  • updated v3.close() such that offTransportClosed runs before apiClient.end()
  • this prevents _onCdpClosed from firing during the API teardown and re-entering close({ force: true })

test plan

  • added unit tests in cdp-connection-close.test.ts which check that:
    • CdpConnection.close() resolves when the websocket is already closed
    • inflight send() calls reject when the websocket is terminated
  • added an e2e test in cdp-connection-close.spec.ts which:
    • terminates the underlying websocket, then calls v3.close() and asserts it resolves
    • terminates the websocket with an inflight CDP call and asserts the pending promise rejects
  • added another e2e test in cdp-close-api-region.spec.ts which:
    • reproduces the full production trigger chain by creating a stagehand instance with disableAPI: false and region: "ap-southeast-1",
    • calls v3.close() and asserts it resolves within 30 seconds

Summary by cubic

Fixes a shutdown race that caused eval runs to hang when using the Stagehand API in non-us-west-2 regions. v3.close() now resolves reliably and inflight CDP requests reject on socket shutdown, addressing Linear STG-1422.

  • Bug Fixes
    • CdpConnection.close(): return immediately if the WebSocket is already CLOSED.
    • Reject inflight CDP calls on "close"/"error" via new CdpConnectionClosedError (exported in public error types).
    • Unhook transport-close handler before apiClient.end() to prevent double-close during API teardown.
    • Added unit and e2e tests (incl. ap-southeast-1) verifying close resolves and inflight requests reject.