feat(mcp): run a test in a session and stop on the failure screen

Merged
KA
19d ago
+1470-107

open_session takes a test, named the way e2e run takes a positional. The session runs that one test and then hands itself over on the screen where the test passed or failed. A coding agent fixing a test gets the failure and the live screen in one call. It doesn't have to click through the flow again from the start, and it can act out the test's next lines from the failure screen before it reruns.

open_session {test: "tests/journey.e2e.ts:4"}
flowchart LR
  A["open_session {test}"] --> B["collect + select<br/>(e2e run's path)"]
  B -- "none / several / unsupported" --> X["NO_TESTS, TEST_AMBIGUOUS,<br/>SESSION_TEST_UNSUPPORTED<br/>(nothing booted)"]
  B --> C["boot engine, open attempt"]
  C --> D["beforeAll, test.extend fixtures,<br/>beforeEach, body<br/>(test timeout)"]
  D --> E["stop: failure evidence,<br/>settle replay cache"]
  E --> F["session: observe, locate, act"]
  F --> G["close_session: afterEach,<br/>fixture teardown, afterAll"]

What changed

  • packages/e2e/src/run/standalone-test.ts (new) selects the test through collect and select, and refuses what a session can't run alone. It runs the hooks, fixtures, and body on the session's attempt under the test's timeout. It publishes the attempt for expect.poll, expect.soft, and test.skip only while test code runs, captures captureFailureEvidence, and settles the replay cache. Teardown waits for close.
  • run/standalone-attempt.ts: a one-shot attempt with a test builds a second fixture set for it on the same session, with its own budget, the test's id and agent context, a cache context as on a run's first attempt, and screenshot markers of its own. After the body the test's budget stays aborted until teardown, so a body its timeout cut short can't drive the session the agent now uses. run/standalone.ts: the attempt's close runs the test's teardown before the host's close cancels its operations.
  • mcp/session.ts: the test input. With a test, the session neither opens the app URL nor launches the pinned app, and the opening text leads with mcp/test-outcome.ts: the outcome, the code and message, the failing line with a code frame, the URL, the nearest nodes, the last 15 steps, and how to rerun. Each session loads the config and the test files in its own module graph (mcp-<session id>), so a rerun sees edited helper files.
  • Test lines (#1317) in a session that ran a test: the test's own steps and opening print none, since they are already in the test file. The first screen after the test is observed through the catalog's context, so the agent's first action on it gets its Test line:, and test_lines (mcp/test-lines.ts) drafts the lines as the ones that continue the test, with no app.open() of the session's own.
  • cli/mcp.ts: an unhandledRejection listener. A rejection while a session's test code runs fails that test, as in a run's worker. One after its body ended is an UNHANDLED_REJECTION line at the newest open test session's close_session, or a warning on stderr once every such session closed. Before any session has run a test, a rejection still shuts the server down, as today.
  • New codes TEST_AMBIGUOUS and SESSION_TEST_UNSUPPORTED. select() takes a noTestsHint, so the MCP NO_TESTS message doesn't name --pass-with-no-tests. codeFrame takes a width.
  • Skill: steps 4 and 5 of SKILL.md's "Workflow: fixing a failed test" (from #1309) name open_session {test} as the way to look live at the failure screen and to rerun, and references/mcp.md gains "Fixing a failing test". Both tell the agent to put the Test line: an action prints in place of a wrong line. Docs: reference/mcp.mdx "Running a test", mcp-server.mdx "Fix a failing test", reference/errors.mdx. Against main, mcp-server.mdx grows from 137 to 161 lines (587 to 712 prose words), reference/mcp.mdx from 499 to 565. No fact was cut. The guide's "Run tests next to a session" lost the sentence the feature made false.

Decisions

  • Replay cache. The test reads and writes it as a run's first attempt: recorded steps replay, a new recording is saved once a later check passes, and a failure evicts what no check confirmed. It settles when the body ends, so nothing the agent does in the session afterwards can confirm or evict an entry. In a session, an afterEach check confirms nothing.
  • Rerun. close_session, then open_session {test} again, not a catalog tool. A rerun needs a fresh attempt, step, catalog, and module graph, which is what an open builds.
  • Refused in v1 with SESSION_TEST_UNSUPPORTED: a serial group member, a test that starts from a setup test's session, a test that runs as several agents. Setup sessions are the biggest gap. Supporting them means running the setup pair through TargetExecutor.runSetupUnit on the session's worker first.
  • One at a time. Test code in different sessions takes turns: publishAttempt has one slot per process. Collection takes turns too: collectModule has one slot, and a file's top-level await can yield it mid-file. An open cancelled while its test waits for its turn runs none of the test's code.
  • A body cut short by its timeout or a stray rejection gets up to cleanupTimeout to reach a harness call, which fails on its aborted budget, before the turn ends.
  • No test result, report, or trace.md for a session's run. Retries and the test's trace and video modes don't apply.

Verified

Ran it locally: yes

After merging main at 6b418575 (with #1316, run_code): test-lines.ts keeps both: run_code statements join test_lines, and after open_session {test} the lines continue that test. The unit tests for both drafts stay. docs/mcp-server.mdx keeps #1316's "How it runs a test line" and this PR's "Fix a failing test", which now says run_code runs a fixed line on the failure screen. The skill keeps open_session {test} and adds #1316's "or run_code for the fixed line" in the fixing workflow and mcp.md step 4, and the test_lines row names both. A new case in mcp-test-run.test.ts runs run_code in a session opened with test and checks the statement continues the test in test_lines. Re-ran pnpm check, 183 unit and 62 integration tests (mcp-test-run, mcp-run-code, mcp-test-lines, mcp-server, mcp-session-host).

After merging main at ec941749 (with #1312 and the agent's test title, #1323/#1344):

  • mcp/server.ts: the instructions keep #1312's "write the test while you walk the flow" and add this PR's pointer to open_session {test}.
  • run/standalone-attempt.ts: the session's attempt takes main's titlePath: []. A follow-up commit gives the test that open_session {test} runs its own title in agent steps, as e2e run does; mcp-test-run.test.ts now checks the first model call's prompt carries Test: increments the counter, and fails with the override removed.
  • references/mcp.md: #1312's intro and "the test is the file you write as you walk", with this PR's clauses on running a test and on the replay cache. errors.mdx: main's NO_TESTS row plus this PR's open_session case.
  • Re-ran pnpm check, skill.test.ts, and the mcp-test-run, mcp-test-lines, mcp-server, mcp-session-host, and open-host-lifecycle tests: all pass.

After merging main at cfe76ba3 (with #1313 and #1314): the skill and docs/mcp-server.mdx conflicted. SKILL.md keeps this PR's step 4 (open_session {test}) before #1314's "Rerunning a test while you fix it", and step 5 reruns that test by file:line or in a new session with open_session {test}. mcp.md step 4 is #1314's one-test rerun with open_session {test} as the first way to reach the failure screen; step 5 is #1313's "leave the session open"; Rules keep this PR's cache sentence and take #1313's rules for running beside a session. "Fixing a failing test" now confirms with npx e2e run tests/<feature>.e2e.ts:<line>. docs/mcp-server.mdx takes #1313's section and says a session can run a test, which uses the replay cache. Re-ran pnpm check, skill.test.ts, and the mcp-test-run, mcp-test-lines, and mcp-server tests on the merge: all pass.

After the rebase onto main (#1317 test lines, #1310 timeout flags, and the rest), re-run on the rebased head:

  • The same MCP smoke on a fresh A/B scratch project with seeds/fix-journey: open_session {test: "tests/journey.e2e.ts"} opened with the same outcome text as below, then the first action on the failure screen printed its line and test_lines drafted only that line:
===== call {"tool":"type","args":{"target":"n32","value":"Grace Hopper"}} (0.1 s)
Typed into #n32.
Test line: await screen.getByRole("textbox", "Full name").fill("Grace Hopper");

===== call {"tool":"test_lines"} (0.0 s)
Test lines from this session's actions after test tests/journey.e2e.ts:4 stopped, in order (1 lines):

await screen.getByRole("textbox", "Full name").fill("Grace Hopper");

Put them in that test where it stopped, in place of the line that failed or after its last line, and add an expect after each step that should change the screen.
  • A session without test on the same server still drafts await app.open(); first, then await app.open("/forms"); for a navigate.
  • tests/integration/mcp-test-run.test.ts (with a new case: a tap by an id from the opening screen prints its line, and test_lines continues the test), #1317's tests/integration/mcp-test-lines.test.ts and tests/unit/mcp-test-lines.test.ts, and tests/unit/skill.test.ts: 43 passed. With dist/mcp/session.js patched back to the branch's pre-rebase observe (not through the catalog's context), the new case fails with No test line: #n7 is not on the newest screen.
  • The other session integration files (mcp-server, mcp-session-host, mcp-user-output): 41 passed. pnpm check passes. Nothing re-run on iOS or Android. On a device, a test session still skips the pinned-app launch, and now records no app.open() line for it either; I checked that path by reading the code only.

Before the rebase:

  • Surface: the built e2e mcp server, driven by an MCP SDK client script, on the A/B harness's scratch project (the testbed playground on a port-0 URL) with the seeded failing test seeds/fix-journey. open_session {test}, then locate, observe, close_session:
===== open_session {"test":"tests/journey.e2e.ts"} (30.8 s)
Test "an admin signs in, saves a profile, and creates a workspace" (tests/journey.e2e.ts:4) failed after 30.32s; 8 steps ran.
LOCATOR_NOT_FOUND: locator matched no nodes within getByLabel("Name")
At tests/journey.e2e.ts:14:3
   12|
   13|   await app.open('/forms');
   14|   await screen.getByLabel('Name').fill('Grace Hopper');
     |   ^
   15|   await screen.getByLabel('Team').selectOption('Mobile');
   16|   await screen.getByLabel('Email notifications').check();
URL at failure: http://127.0.0.1:50536/forms
On screen, closest to what the locator asked for:
  #n31 text="Full name"
  #n32 textbox "Full name" purpose=username
Steps:
  1. ✓ app.open /login (tests/journey.e2e.ts:6)
  ...
  7. ✓ app.open /forms (tests/journey.e2e.ts:13)
  8. ✗ locator.fill getByLabel("Name") LOCATOR_NOT_FOUND (tests/journey.e2e.ts:14)
The session is on the screen where the test stopped, shown below: observe and locate from here, and act out the test's next lines to check them too, then fix the test file. Its afterEach hooks and fixture teardown run at close_session. To run the fixed test from the start, close_session, then open_session {test: "tests/journey.e2e.ts"}.

Session 01a15e7a-... open on target "web" (platform web, engine web 0.15.2), headless; config .../e2e.config.ts.
...
Current screen (revision b3, path /forms, 45 nodes):
...
 #n31 text="Full name"
 #n32 textbox "Full name" purpose=username

===== call {"tool":"locate","args":{"label":"Full name"}} (0.0 s)
1 node matches getByLabel("Full name").
Use: screen.getByLabel("Full name")

===== call {"tool":"observe"} (0.1 s)
Current screen (revision b5, path /forms, 45 nodes):
...
  • The same project with the fixed test (seeds/fixed-journey) and an afterEach added: the test passed in 427 ms and the session sat on /wizard. The afterEach ran only at close_session, and a failing one showed as Cleanup (afterEach): LOCATOR_NOT_FOUND: ... in its result.
  • Selection errors on the same server, before anything booted:
NO_TESTS: no test file matched nope; the config globs discovered tests/auth.setup.e2e.ts, tests/journey.e2e.ts, tests/many.e2e.ts; name a test file the config finds, with :line for one test in it
TEST_AMBIGUOUS: "many" selects 5 tests on target "web", and a session runs one; name it as file:line: tests/many.e2e.ts:3 "first", tests/many.e2e.ts:7 "second", tests/many.e2e.ts:12 "flow › step one", tests/many.e2e.ts:15 "flow › step two", tests/many.e2e.ts:20 "signed in"
SESSION_TEST_UNSUPPORTED: test "flow › step one" (tests/many.e2e.ts:12) is in a serial group, whose tests run in order on one shared attempt; a session runs a test alone, so run the group with npx e2e run
SESSION_TEST_UNSUPPORTED: test "signed in" (tests/many.e2e.ts:20) starts from session "admin", which a setup test saves first; a session does not run setup tests yet, so run it with npx e2e run
  • tests/integration/mcp-test-run.test.ts, over the built CLI and an MCP client: a pass on its final screen with afterEach at close, a failure with its line, frame, and nearest nodes and an observe after it, a rerun that sees an edited helper, a forgotten await that fails the test and leaves the server serving, a late rejection reported at close and one after close as a warning, a body past its timeout that can't navigate the session, a beforeAll failure reported once, a screenshot baseline that passes in the next session, and the selection errors. In-process: two sessions that collect and run side by side, an open cancelled while queued that runs none of its test, and a session's recording that replays in the next session and in e2e run with no model call. The side-by-side, cancelled, and timed-out cases each fail when the fix they cover is removed from dist.
  • The testbed's own tests/forms.e2e.ts:9 (a describe with a beforeEach) passes in a session, and tests/forms.e2e.ts alone is TEST_AMBIGUOUS listing its five tests. The testbed's forms, wizard, and smoke files still pass under e2e run.

A/B: a fresh agent fixing the seeded test

run-arm.mjs, Sonnet 5.5, the seeded test with three mistakes that surface one per run, each failing run waiting out the 30 s action timeout. Main baselines are the harness's own (main-fix-*) and PR B's main arms of the no-source variant (b-main-fixnosrc-*). Arms 1 to 4 ran before #1309 landed, with the new text as a step 6 of the writing workflow: f1-fix-1 and -2 with a first wording, the rest with two later ones that differ only in how hard the step pushes toward a session. Arms 5 ran the text as it sits now inside #1309's fix workflow, before the last wording fix on which line names a test.

RunTaskWall mine2e runsSessionsMCP callsTool callsModel stepsCost $Final exit
main-fix-1fix1.740014120.100
main-fix-2fix1.120014110.060
f1-fix-1fix1.54001170.100
f1-fix-2fix1.64001170.080
f1-fix-3fix1.521615110.130
f1-fix-4fix2.131716120.110
f1-fix-5fix1.13001390.100
b-main-fixnosrc-1no source1.64001170.080
b-main-fixnosrc-2no source1.54001060.080
f1-fixnosrc-1no source1.54001060.080
f1-fixnosrc-2no source2.131715100.100
f1-fixnosrc-3no source1.54001060.080
f1-fixnosrc-4no source1.54001070.080
f1-fixnosrc-5no source1.54111180.100
  • No speedup shows. On this small app Sonnet fixes each mistake from the terminal's nearest nodes and the trace page, and 6 of the 10 branch arms never opened a session. main-fix-2's and f1-fix-5's low run counts came from reading the app's source outside the project, which the no-source task forbids.
  • The 3 arms that opened a session did what the feature is for. In f1-fix-3, after one CLI run and the first fix, the agent opened {test: "tests/journey.e2e.ts"}. That landed on the wizard's wrong button, and the agent tapped Next, picked Pro, tapped Next and Create workspace, and read the summary, all in the session. It fixed the last two mistakes together, and its one final e2e run passed: 2 runs instead of 4. f1-fix-4 and f1-fixnosrc-2 did the same after a second CLI run, so the session paid the wizard's 30 s timeout a second time.
  • f1-fixnosrc-5 passed test: "tests/journey.e2e.ts:23", the wizard line it wanted to look at. That is NO_TESTS (:23 names no test (declared at line 4)), and the agent went back to the CLI. The last commit says in the tool description, the NO_TESTS hint, the skill, and the reference that the line is the test( call's, and that a file holding one test needs none. Accepting a line inside a test would be friendlier, but it departs from e2e run's positional rules, so I left that for the maintainer.
  • The session pays the same action timeout as a run. What it saves is the failing rerun per remaining mistake, and the walk from the start over MCP the maintainer described. Neither shows much on a small app where the CLI output names the fix.

Review

Two fresh-context review rounds before opening. Fixed: a forgotten await that crashed the server, screenshot markers that outlived a session, concurrent collections that clashed, a cancelled open that still ran its hooks, a beforeAll failure reported twice, stale comments and docs (including docs/cache.mdx), a late rejection that could still crash the server, a timed-out body that kept driving the session, and a rejection after the verdict that flipped a passed outcome. Declined: extracting a shared body runner out of executeAttempt (below).

For the maintainer

  • standalone-test.ts repeats the body part of executeAttempt: beforeEach, the body race, steps not awaited, soft failures, settling, the teardown loop, and cache settlement. Sharing it means pulling a body and teardown runner out of executeAttempt, with "stop before teardown" and "budget goes back to the session" as options. That is a refactor of the run path, so I left it for its own PR.
  • Tests that start from a setup test's session are refused. That's the biggest gap for apps that sign in through a setup test.
  • Whether open_session {test: "file:<line inside a test>"} should pick the test around that line. Agents pass the failing line; e2e run doesn't accept it either.