feat(web)!: WebDriver BiDi driver replaces Playwright
Draft
OK
10d ago+21826+21826+01234567890123456789012345678901234567890123456789-13555-13555-01234567890123456789012345678901234567890123456789
What and why
- Experimental WebDriver BiDi driver for
@e2e-dev/web:web({ driver: 'bidi' }), orE2E_WEB_DRIVER=bidifor a whole run. Chromium and Firefox. Playwright stays the default. - Own thin BiDi client over Node's
WebSocket, typed withwebdriver-bidi-protocol. Firefox speaks BiDi natively. Chromium runs thechromium-bidimapper in a child process. No chromedriver or geckodriver. Both drivers launch Playwright's binaries with Playwright's flags and prefs. - Same engine contract and in-page reader as the Playwright path. Locate runs the contract's
resolveExpressionover node-mode reads.resolveExpressiongains optionalmatchesandhasTexthooks so the driver locates by Playwright's text, label, role, and hidden rules (changeset fore2e;@e2e-dev/webpeers one2e >=0.22.1). - Parity with the Playwright driver's options and
browserfixture on Chromium. Not built: a persistent remote context (reconnectEndpoint, provider scopeattempt). Documented limits: Playwright selector chains, rotated or mirrored frames. - Code in
packages/web/src/bidi/(surface, session, events, steering, observe, actions, network, state, artifacts, video, drag, launch). Mapping, gaps with upstream links, methodology, and review history:packages/web/bench/bidi-benchmark.md. - CI: the testbed on BiDi Chromium and Firefox, and the parity suite's BiDi entries, are part of the spec gate.
- Stacked on #1505 (the Playwright parity suite on the Playwright driver); this PR only adds BiDi matrix entries and
bidiGapskips to it.
Correctness
| Suite | PW chromium | BiDi chromium | PW firefox | BiDi firefox |
|---|---|---|---|---|
| testbed (83) | 82 + 1 skip | 82 + 1 skip | 81, 1 fail | 79 + 1 skip (3 left out) |
| web benchmark (91) | 91 | 91 | 91 | 89, 2 fail |
The remaining failures:
- Playwright Firefox raw mouse drag: also fails on main (Firefox rounds y to 19).
- BiDi Firefox HTML5 drag and drop (2 + 2): Firefox's remote agent never synthesizes
drop(bug 1515879). - BiDi Firefox
emulateMedia:setMediaFeaturesOverridelands in Firefox 159 (bug 2064354).
Parity with Playwright's own tests
#1505's suite (511 tests from Playwright's tests/page at v1.64.1), all four combos, at 30f1ce3b4:
| Driver | Chromium | Firefox |
|---|---|---|
| Playwright | 486 pass, 25 skip | 477 pass, 34 skip |
| BiDi | 473 pass, 38 skip | 470 pass, 41 skip |
- No failures. BiDi-only skips are browser or protocol gaps, each linked: most are in Playwright's own BiDi expectations (
bidi-chromium-*.txt,moz-firefox-nightly-page.txt). The rest: WebDriver numpad keys are NumLock-on and it has no media keys; Chromium aborts a downloading navigation before announcing it; macOS editing bindings and Firefox's Option layer (macOS-only). - About 40 BiDi bugs the suite found are fixed here: locate rules, scrollIntoView, keys and error messages, shadow-root CSS, header overrides and redirects, CORS preflight, fulfill bodies, upload checks, a Chromium navigation POST hang, Firefox navigation downloads. The full list is in the benchmark doc.
Benchmark totals (20 warm / 10 cold iterations, drivers interleaved, M5 Pro, headless)
| Total | PW chromium | BiDi chromium | PW firefox | BiDi firefox | BiDi vs PW chromium | BiDi vs PW firefox |
|---|---|---|---|---|---|---|
| sum of p50s, every scenario (ms) | 1534 | 981 | 2623 | 2116 | -36% | -19% |
| sum of means, every scenario (ms) | 1566 | 1021 | 2697 | 2135 | -35% | -21% |
| realistic test: cold launch + new page + 6 agent steps (ms) | 334 | 274 | 768 | 1279 | -18% | +66% |
| testbed wall clock, median of 3 (ms) | 17525 | 16895 | 21102 | 20980 | -4% | -1% |
| node CPU, warm loop (s) | 18.5 | 1.4 | 18.0 | 2.4 | -92% | -87% |
| node + browser CPU, warm loop (s, floor) | 51.7 | 20.3 | 77.7 | 33.7 | -61% | -57% |
| peak RSS, node + browser (MB) | 1405 | 1542 | 2351 | 2644 | +10% | +12% |
- N = 6 agent steps: the testbed averages 6.25 steps per test (median 5).
- Firefox's realistic test is slower because its BiDi cold launch takes 925 ms vs 314 ms: the Remote Agent holds
session.newuntil startup finishes. A worker pays it once. - Per-scenario p50s with bootstrap 95% intervals are in the benchmark doc. Highlights: observe + pixels 0.23-0.35x, locate 0.35-0.62x, agent flow 0.64-0.65x, heavy-page Chromium screenshot 1.25x. Rerun after the parity fixes: every total within noise of the previous run.
Review
Nine rounds of line-by-line review of the full diff, each fixed in focused commits: 38/15/1, 2/24/2, 0/15/1, 0/10/3, 0/6/0, 1/4/0, 0/7/0, 0/8/1, 0/7/0 (Critical/Important/Nit). All fixed. Rounds 7-9 had no Critical findings, but none came back at zero Important. The round-9 fixes were verified by the testbed and the parity suite; the 3-round cap left them without another review.
Issue
Spike, no issue.
Verified
Ran it locally: yes
- Built packages, real CLI: the testbed and the parity suite on all four combos at 30f1ce3b4; the web benchmark suite table is from c6fcde8f8 (not rerun after the parity fixes).
- Scratch checks on both browsers against the built engine: saved sessions across two origins with IndexedDB (no app requests during restore), same-URL history back, masked screenshots under hostile page CSS, a modal reopening every 5 ms and a scaled modal, clicks inside a scaled iframe, labels / shadow-root CSS / frames in shadow roots identical to the Playwright driver, video length matching wall time.
@e2e-dev/webunit tests 381 pass,e2eunit tests pass; build, typecheck, oxlint, fallow, docs, plugin, peer-range, and changeset checks are clean.
Checklist
- Changeset added (
pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only. - Breaking change: title carries
!, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md). - Docs updated in the same PR: the
docs/**/*.mdxpage for the behavior, andskills/e2e/if the skill describes it. - Contract change: emitted
.d.tsreviewed,tests/types/sdk-types.tsupdated; wire change edits the schema and both fixtures. - Engine contract change (
e2e/engine): changesets fore2e,@e2e-dev/web, and@e2e-dev/mobile.