feat(web)!: WebDriver BiDi driver replaces Playwright

Draft
OK
10d ago
+21826-13555

What and why

  • Experimental WebDriver BiDi driver for @e2e-dev/web: web({ driver: 'bidi' }), or E2E_WEB_DRIVER=bidi for a whole run. Chromium and Firefox. Playwright stays the default.
  • Own thin BiDi client over Node's WebSocket, typed with webdriver-bidi-protocol. Firefox speaks BiDi natively. Chromium runs the chromium-bidi mapper in a child process. No chromedriver or geckodriver. Both drivers launch Playwright's binaries with Playwright's flags and prefs.
  • Same engine contract and in-page reader as the Playwright path. Locate runs the contract's resolveExpression over node-mode reads. resolveExpression gains optional matches and hasText hooks so the driver locates by Playwright's text, label, role, and hidden rules (changeset for e2e; @e2e-dev/web peers on e2e >=0.22.1).
  • Parity with the Playwright driver's options and browser fixture on Chromium. Not built: a persistent remote context (reconnectEndpoint, provider scope attempt). Documented limits: Playwright selector chains, rotated or mirrored frames.
  • Code in packages/web/src/bidi/ (surface, session, events, steering, observe, actions, network, state, artifacts, video, drag, launch). Mapping, gaps with upstream links, methodology, and review history: packages/web/bench/bidi-benchmark.md.
  • CI: the testbed on BiDi Chromium and Firefox, and the parity suite's BiDi entries, are part of the spec gate.
  • Stacked on #1505 (the Playwright parity suite on the Playwright driver); this PR only adds BiDi matrix entries and bidiGap skips to it.

Correctness

SuitePW chromiumBiDi chromiumPW firefoxBiDi firefox
testbed (83)82 + 1 skip82 + 1 skip81, 1 fail79 + 1 skip (3 left out)
web benchmark (91)91919189, 2 fail

The remaining failures:

  • Playwright Firefox raw mouse drag: also fails on main (Firefox rounds y to 19).
  • BiDi Firefox HTML5 drag and drop (2 + 2): Firefox's remote agent never synthesizes drop (bug 1515879).
  • BiDi Firefox emulateMedia: setMediaFeaturesOverride lands in Firefox 159 (bug 2064354).

Parity with Playwright's own tests

#1505's suite (511 tests from Playwright's tests/page at v1.64.1), all four combos, at 30f1ce3b4:

DriverChromiumFirefox
Playwright486 pass, 25 skip477 pass, 34 skip
BiDi473 pass, 38 skip470 pass, 41 skip
  • No failures. BiDi-only skips are browser or protocol gaps, each linked: most are in Playwright's own BiDi expectations (bidi-chromium-*.txt, moz-firefox-nightly-page.txt). The rest: WebDriver numpad keys are NumLock-on and it has no media keys; Chromium aborts a downloading navigation before announcing it; macOS editing bindings and Firefox's Option layer (macOS-only).
  • About 40 BiDi bugs the suite found are fixed here: locate rules, scrollIntoView, keys and error messages, shadow-root CSS, header overrides and redirects, CORS preflight, fulfill bodies, upload checks, a Chromium navigation POST hang, Firefox navigation downloads. The full list is in the benchmark doc.

Benchmark totals (20 warm / 10 cold iterations, drivers interleaved, M5 Pro, headless)

TotalPW chromiumBiDi chromiumPW firefoxBiDi firefoxBiDi vs PW chromiumBiDi vs PW firefox
sum of p50s, every scenario (ms)153498126232116-36%-19%
sum of means, every scenario (ms)1566102126972135-35%-21%
realistic test: cold launch + new page + 6 agent steps (ms)3342747681279-18%+66%
testbed wall clock, median of 3 (ms)17525168952110220980-4%-1%
node CPU, warm loop (s)18.51.418.02.4-92%-87%
node + browser CPU, warm loop (s, floor)51.720.377.733.7-61%-57%
peak RSS, node + browser (MB)1405154223512644+10%+12%
  • N = 6 agent steps: the testbed averages 6.25 steps per test (median 5).
  • Firefox's realistic test is slower because its BiDi cold launch takes 925 ms vs 314 ms: the Remote Agent holds session.new until startup finishes. A worker pays it once.
  • Per-scenario p50s with bootstrap 95% intervals are in the benchmark doc. Highlights: observe + pixels 0.23-0.35x, locate 0.35-0.62x, agent flow 0.64-0.65x, heavy-page Chromium screenshot 1.25x. Rerun after the parity fixes: every total within noise of the previous run.

Review

Nine rounds of line-by-line review of the full diff, each fixed in focused commits: 38/15/1, 2/24/2, 0/15/1, 0/10/3, 0/6/0, 1/4/0, 0/7/0, 0/8/1, 0/7/0 (Critical/Important/Nit). All fixed. Rounds 7-9 had no Critical findings, but none came back at zero Important. The round-9 fixes were verified by the testbed and the parity suite; the 3-round cap left them without another review.

Issue

Spike, no issue.

Verified

Ran it locally: yes

  • Built packages, real CLI: the testbed and the parity suite on all four combos at 30f1ce3b4; the web benchmark suite table is from c6fcde8f8 (not rerun after the parity fixes).
  • Scratch checks on both browsers against the built engine: saved sessions across two origins with IndexedDB (no app requests during restore), same-URL history back, masked screenshots under hostile page CSS, a modal reopening every 5 ms and a scaled modal, clicks inside a scaled iframe, labels / shadow-root CSS / frames in shadow roots identical to the Playwright driver, video length matching wall time.
  • @e2e-dev/web unit tests 381 pass, e2e unit tests pass; build, typecheck, oxlint, fallow, docs, plugin, peer-range, and changeset checks are clean.

Checklist

  • Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
  • Breaking change: title carries !, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md).
  • Docs updated in the same PR: the docs/**/*.mdx page for the behavior, and skills/e2e/ if the skill describes it.
  • Contract change: emitted .d.ts reviewed, tests/types/sdk-types.ts updated; wire change edits the schema and both fixtures.
  • Engine contract change (e2e/engine): changesets for e2e, @e2e-dev/web, and @e2e-dev/mobile.