fix(web): answer native dialogs an agent step's action opens

OK
10d ago
+549-43

An agent never sees a native alert, confirm, prompt, or leave-page dialog: they sit outside the page. With no browser.onDialog handler, the web engine dismissed each one and failed the next operation with INVALID_STATE. So an agent step could not leave a form with unsaved changes or confirm a delete, live or from the replay cache.

  • DialogRouter asks its surface whether an agent step made the newest action (packages/web/src/dialogs.ts). If so, a dialog no handler takes is answered: alerts, confirms, and leave-page dialogs are accepted, and a prompt is dismissed, since the agent never sees its text. The app log gets one system line, such as confirm dialog "Delete this label?" accepted for the agent.
  • Ownership moves only on actions. PlaywrightSurface.guardAction() (packages/web/src/surface.ts) wraps every operation that acts: navigation, input, browser.evaluate, keyboard, mouse, browser.setViewport, browser.emulateMedia. It claims the dialog once the operation runs, so one refused by a pending error claims nothing. Reads (observe, locate, screenshots) and a route handler's route.fetch never take a dialog. So a test's own late dialog still fails its next step while an agent judgment polls, and a test route handler fetching mid-action can't steal the agent's dialog.
  • OperationContext.agentStep (new, optional) marks an agent step's operations, replays included (packages/e2e/src/agent/phases.ts). Replayed actions keep origin: 'test' for settling, so without it a recorded dialog step failed under --strict-cache. It crosses the engine protocol. An older core omits it and the engine falls back to origin.
  • A dialog a test's own step opens behaves as before: dismissed, and the next step fails with INVALID_STATE. A registered handler always wins.
  • docs/reference/web.mdx said an unhandled dialog throws APP_NOT_OPEN. The code throws INVALID_STATE, so the table now says that.
  • The web benchmark gets an agent scenario on /e/deferred-dialog, whose confirm opens 900 ms after the click, with its committed recording.

Why

In 201 e2e explore runs on 0.22.0 against a self-hosted Gitea (their report.json and AI traces):

  • 226 agent observations and actions failed with unhandled beforeunload dialog.
  • 114 navigate calls failed with net::ERR_ABORTED. 113 of them were the dismissed leave-page dialog.
  • 59 steps ended blocked. The model saw only Action failed: unhandled beforeunload dialog: and retried back until the step timed out.

For review

  • Confirms are accepted. The agent's action asked for the destructive step and the agent can't see the dialog to decline it. That also removes the app's last guard if an explorer clicks "Delete workspace" against its instructions. The alternative is to dismiss confirms, tell the model the text, and give it an explicit way to accept, which needs a model-visible notice channel first.
  • The model and MCP callers don't see the app log line. An MCP action's test line registers no handler, so the docs and the MCP skill now say a scripted test of that flow needs browser.onDialog. run_code is test code, so its dialogs still fail the next call. Surfacing answered dialogs in action results is a follow-up.
  • Porting to #1503: a trial cherry-pick onto its head conflicts only in driver/dialogs.ts imports. BidiSession builds the router there and would pass the same two callbacks, and its action entry points would take guardAction's ownership line.

Verified

Ran it locally: yes

  • packages/web/tests/integration/agent-dialogs.test.ts, real Chromium, 8 cases: an agent leaves a dirty form; agent confirm, alert, and prompt; a replayed agent action (origin: 'test', agentStep: true); a test route handler's route.fetch before the agent's alert; a test's setViewport whose resize handler confirms after an agent tap, which must fail the test's next step; a test action refused by a pending route error, which must leave the agent's alert with the agent; a test's 300 ms delayed confirm while agent reads poll, which must still fail with INVALID_STATE; and a test's own confirm. On main, both live agent cases fail with unhandled ... dialog. The viewport and refused-action cases fail without their fixes.
  • Web benchmark, tests-agent deferred-dialog: recorded live once, then replayed twice with CI=1 --strict-cache: Cache 1 replayed, no model call, and the app log shows confirm dialog "Really submit the report?" accepted for the agent.
  • The whole web benchmark agent suite under --strict-cache, with the same counts on main for the existing recordings:
mainthis branch
Tests33 passed34 passed (+ deferred-dialog)
Cache26 replayed · 2 handed off · 4 missed27 replayed · 2 handed off · 4 missed
  • Testbed tests/dialog.e2e.ts and tests/dialogs-frames.e2e.ts (15 tests) and web benchmark tests/native-dialogs.e2e.ts and tests/deferred-dialog.e2e.ts (4 tests) pass, so scripted handlers behave as before.
  • e2e explore (openai/gpt-6-luna-fast, --max-steps 4) on a scratch notes app with a leave-page guard on its editor and a confirm() before each delete. Its delete removes the wrong note, a planted bug:
mainthis branch
Steps3 of 4, all blocked, ended "steps kept failing"4 of 4, the step limit was reached
Failed agent actions7 (unhandled beforeunload dialog)0
Planted delete bugnot reachedfound: "Deleting Trip plan removes Groceries instead"
Cost$0.0107$0.0106
  • Agentic repro tests against the same app (e2e run --tag bugbash):
Repromainthis branch
Delete removes the chosen note (real bug)AUTOMATION_UNSUPPORTED: blocked by the confirmASSERTION_FAILED: confirmed
Leaving the editor unsaved keeps the list (not a bug)ACTION_FAILED: blocked by the leave-page dialogpasses