feat(runner): host e2e sessions in another test runner

Merged
BH
14d ago
+2375-288

Projects on Playwright Test, Vitest, Jest, or another harness can use e2e engines without moving their suites to e2e run.

  • openHost<Fixtures>() on e2e/runner owns the worker lifetime: prepare, app processes, engine init, once per worker. host.openSession({ name, timeout, signal, agent, onStep }) opens one test-lifetime attempt: fresh fixtures, deadline, steps, artifacts, and a trace.md at close.
  • A hosted attempt is published, so expect.soft collects for the session and expect.poll honors its budget and cancellation. One hosted attempt may be opening, live, or closing per process.
  • The session timeout is enforced for every operation: when it passes, running steps fail with TEST_TIMEOUT, the session is timed out, and close() reports it. An engine that ignores its operation timeout, or a model call, cannot outlive it.
  • A launch that fails or is cancelled leaves the host usable, as e2e run keeps its worker for the retry. Only failed engine cleanup retires a host.
  • Artifacts land under results/host/<safe segment>/; the session name never becomes a raw path component. Opening errors, notices, titles, and traces are redacted.
  • Both lifetimes have idempotent close() and async disposal. Separate hosts and config loads cannot bind secrets or engine ownership to one another.
  • The MCP one-shot path stays internal and shares the worker (packages/e2e/src/run/standalone.ts), the attempt (standalone-attempt.ts), and target selection (select-target.ts). No new dependencies, engine contract, or CI changes.
const test = base.extend<{ e2e: E2ESession<Fixtures> }, { e2eHost: E2EHost<Fixtures> }>({
  e2eHost: [async ({}, use) => {
    const host = await openHost<Fixtures>({ target: 'android' });
    try { await use(host); } finally { await host.close(); }
  }, { scope: 'worker' }],
  e2e: async ({ e2eHost }, use, testInfo) => {
    const session = await e2eHost.openSession({ name: testInfo.titlePath.join(' › '), timeout: testInfo.timeout || undefined });
    try { await use(session); } finally {
      const failures = await session.close();
      await testInfo.attach('e2e trace', { path: session.tracePath, contentType: 'text/markdown' });
      if (failures.length > 0) throw new AggregateError(failures, 'e2e session close failed');
    }
  },
});
flowchart LR
  W[Worker fixture] -->|openHost: prepare, processes, init once| H[Prepared host]
  H -->|openSession with test name| S[Reserve, publish, launch attempt]
  S --> A[Fixtures, soft expectations, bounded polling, deadline timer]
  A -->|session.close| C[End attempt, write trace, release publication]
  C -->|next test| H
  H -->|host.close| D[Dispose engine, finish, stop processes]

Verified

Ran it locally: yes

  • A scratch Playwright Test project (@playwright/test 1.64.0) hosting the testbed web target in real Chromium through the built dist, four tests on one worker: profile form filled; fresh form state on the reused host (same runId, one host boot of ~500 ms, sessions open in under 10 ms); expect.soft failure returned by close() and failing the Playwright test; a tap() on a missing element with the session timeout at 1500 ms settled at 1463 ms with TEST_TIMEOUT, the trace page reading timed out.
[host] opened in 492ms runId=01a13158-795c-7936-8579-70ca501493b7
[session] "fills a profile form" opened in 9ms ... closed, failures=[]
[session] "starts with fresh form state on the reused host" opened in 3ms ... closed, failures=[]
[session] "collects a soft failure and fails the test at close" closed, failures=["ASSERTION_FAILED: 1 soft assertion failed ..."]
  [step] locator.tap getByLabel("No such field"): failed TEST_TIMEOUT
  [deadline] tap settled after 1463ms code=TEST_TIMEOUT message=session timed out after 1500 ms
[host] closed, failures=[]
  4 passed (3.5s)
  • Fake-engine coverage in packages/e2e/tests/integration/open-host-*.test.ts: lifecycle and reservation races, deadline (polls, contributed ops, a hung core op), redaction, engine ownership, trace pages.
  • Your runner owns retries and test-body failures; e2e writes per-session traces, not a run report. The session timeout starts after launch and does not bound provisioning or disposal.