feat(mcp): run a test in a session and stop on the failure screen
open_session takes a test, named the way e2e run takes a positional. The session runs that one test and then hands itself over on the screen where the test passed or failed. A coding agent fixing a test gets the failure and the live screen in one call. It doesn't have to click through the flow again from the start, and it can act out the test's next lines from the failure screen before it reruns.
open_session {test: "tests/journey.e2e.ts:4"}
flowchart LR
A["open_session {test}"] --> B["collect + select<br/>(e2e run's path)"]
B -- "none / several / unsupported" --> X["NO_TESTS, TEST_AMBIGUOUS,<br/>SESSION_TEST_UNSUPPORTED<br/>(nothing booted)"]
B --> C["boot engine, open attempt"]
C --> D["beforeAll, test.extend fixtures,<br/>beforeEach, body<br/>(test timeout)"]
D --> E["stop: failure evidence,<br/>settle replay cache"]
E --> F["session: observe, locate, act"]
F --> G["close_session: afterEach,<br/>fixture teardown, afterAll"]
What changed
packages/e2e/src/run/standalone-test.ts(new) selects the test throughcollectandselect, and refuses what a session can't run alone. It runs the hooks, fixtures, and body on the session's attempt under the test's timeout. It publishes the attempt forexpect.poll,expect.soft, andtest.skiponly while test code runs, capturescaptureFailureEvidence, and settles the replay cache. Teardown waits for close.run/standalone-attempt.ts: a one-shot attempt with atestbuilds a second fixture set for it on the same session, with its own budget, the test's id and agent context, a cache context as on a run's first attempt, and screenshot markers of its own. After the body the test's budget stays aborted until teardown, so a body its timeout cut short can't drive the session the agent now uses.run/standalone.ts: the attempt's close runs the test's teardown before the host's close cancels its operations.mcp/session.ts: thetestinput. With a test, the session neither opens the app URL nor launches the pinned app, and the opening text leads withmcp/test-outcome.ts: the outcome, the code and message, the failing line with a code frame, the URL, the nearest nodes, the last 15 steps, and how to rerun. Each session loads the config and the test files in its own module graph (mcp-<session id>), so a rerun sees edited helper files.- Test lines (#1317) in a session that ran a test: the test's own steps and opening print none, since they are already in the test file. The first screen after the test is observed through the catalog's context, so the agent's first action on it gets its
Test line:, andtest_lines(mcp/test-lines.ts) drafts the lines as the ones that continue the test, with noapp.open()of the session's own. cli/mcp.ts: anunhandledRejectionlistener. A rejection while a session's test code runs fails that test, as in a run's worker. One after its body ended is anUNHANDLED_REJECTIONline at the newest open test session'sclose_session, or a warning on stderr once every such session closed. Before any session has run a test, a rejection still shuts the server down, as today.- New codes
TEST_AMBIGUOUSandSESSION_TEST_UNSUPPORTED.select()takes anoTestsHint, so the MCPNO_TESTSmessage doesn't name--pass-with-no-tests.codeFrametakes a width. - Skill: steps 4 and 5 of
SKILL.md's "Workflow: fixing a failed test" (from #1309) nameopen_session {test}as the way to look live at the failure screen and to rerun, andreferences/mcp.mdgains "Fixing a failing test". Both tell the agent to put theTest line:an action prints in place of a wrong line. Docs:reference/mcp.mdx"Running a test",mcp-server.mdx"Fix a failing test",reference/errors.mdx. Against main,mcp-server.mdxgrows from 137 to 161 lines (587 to 712 prose words),reference/mcp.mdxfrom 499 to 565. No fact was cut. The guide's "Run tests next to a session" lost the sentence the feature made false.
Decisions
- Replay cache. The test reads and writes it as a run's first attempt: recorded steps replay, a new recording is saved once a later check passes, and a failure evicts what no check confirmed. It settles when the body ends, so nothing the agent does in the session afterwards can confirm or evict an entry. In a session, an
afterEachcheck confirms nothing. - Rerun.
close_session, thenopen_session {test}again, not a catalog tool. A rerun needs a fresh attempt, step, catalog, and module graph, which is what an open builds. - Refused in v1 with
SESSION_TEST_UNSUPPORTED: a serial group member, a test that starts from a setup test'ssession, a test that runs as several agents. Setup sessions are the biggest gap. Supporting them means running the setup pair throughTargetExecutor.runSetupUniton the session's worker first. - One at a time. Test code in different sessions takes turns:
publishAttempthas one slot per process. Collection takes turns too:collectModulehas one slot, and a file's top-levelawaitcan yield it mid-file. An open cancelled while its test waits for its turn runs none of the test's code. - A body cut short by its timeout or a stray rejection gets up to
cleanupTimeoutto reach a harness call, which fails on its aborted budget, before the turn ends. - No test result, report, or
trace.mdfor a session's run. Retries and the test'straceandvideomodes don't apply.
Verified
Ran it locally: yes
After merging main at 6b418575 (with #1316, run_code): test-lines.ts keeps both: run_code statements join test_lines, and after open_session {test} the lines continue that test. The unit tests for both drafts stay. docs/mcp-server.mdx keeps #1316's "How it runs a test line" and this PR's "Fix a failing test", which now says run_code runs a fixed line on the failure screen. The skill keeps open_session {test} and adds #1316's "or run_code for the fixed line" in the fixing workflow and mcp.md step 4, and the test_lines row names both. A new case in mcp-test-run.test.ts runs run_code in a session opened with test and checks the statement continues the test in test_lines. Re-ran pnpm check, 183 unit and 62 integration tests (mcp-test-run, mcp-run-code, mcp-test-lines, mcp-server, mcp-session-host).
After merging main at ec941749 (with #1312 and the agent's test title, #1323/#1344):
mcp/server.ts: the instructions keep #1312's "write the test while you walk the flow" and add this PR's pointer toopen_session {test}.run/standalone-attempt.ts: the session's attempt takes main'stitlePath: []. A follow-up commit gives the test thatopen_session {test}runs its own title in agent steps, ase2e rundoes;mcp-test-run.test.tsnow checks the first model call's prompt carriesTest: increments the counter, and fails with the override removed.references/mcp.md: #1312's intro and "the test is the file you write as you walk", with this PR's clauses on running a test and on the replay cache.errors.mdx: main'sNO_TESTSrow plus this PR'sopen_sessioncase.- Re-ran
pnpm check,skill.test.ts, and themcp-test-run,mcp-test-lines,mcp-server,mcp-session-host, andopen-host-lifecycletests: all pass.
After merging main at cfe76ba3 (with #1313 and #1314): the skill and docs/mcp-server.mdx conflicted. SKILL.md keeps this PR's step 4 (open_session {test}) before #1314's "Rerunning a test while you fix it", and step 5 reruns that test by file:line or in a new session with open_session {test}. mcp.md step 4 is #1314's one-test rerun with open_session {test} as the first way to reach the failure screen; step 5 is #1313's "leave the session open"; Rules keep this PR's cache sentence and take #1313's rules for running beside a session. "Fixing a failing test" now confirms with npx e2e run tests/<feature>.e2e.ts:<line>. docs/mcp-server.mdx takes #1313's section and says a session can run a test, which uses the replay cache. Re-ran pnpm check, skill.test.ts, and the mcp-test-run, mcp-test-lines, and mcp-server tests on the merge: all pass.
After the rebase onto main (#1317 test lines, #1310 timeout flags, and the rest), re-run on the rebased head:
- The same MCP smoke on a fresh A/B scratch project with
seeds/fix-journey:open_session {test: "tests/journey.e2e.ts"}opened with the same outcome text as below, then the first action on the failure screen printed its line andtest_linesdrafted only that line:
===== call {"tool":"type","args":{"target":"n32","value":"Grace Hopper"}} (0.1 s)
Typed into #n32.
Test line: await screen.getByRole("textbox", "Full name").fill("Grace Hopper");
===== call {"tool":"test_lines"} (0.0 s)
Test lines from this session's actions after test tests/journey.e2e.ts:4 stopped, in order (1 lines):
await screen.getByRole("textbox", "Full name").fill("Grace Hopper");
Put them in that test where it stopped, in place of the line that failed or after its last line, and add an expect after each step that should change the screen.
- A session without
teston the same server still draftsawait app.open();first, thenawait app.open("/forms");for anavigate. tests/integration/mcp-test-run.test.ts(with a new case: a tap by an id from the opening screen prints its line, andtest_linescontinues the test), #1317'stests/integration/mcp-test-lines.test.tsandtests/unit/mcp-test-lines.test.ts, andtests/unit/skill.test.ts: 43 passed. Withdist/mcp/session.jspatched back to the branch's pre-rebase observe (not through the catalog's context), the new case fails withNo test line: #n7 is not on the newest screen.- The other session integration files (
mcp-server,mcp-session-host,mcp-user-output): 41 passed.pnpm checkpasses. Nothing re-run on iOS or Android. On a device, a test session still skips the pinned-app launch, and now records noapp.open()line for it either; I checked that path by reading the code only.
Before the rebase:
- Surface: the built
e2e mcpserver, driven by an MCP SDK client script, on the A/B harness's scratch project (the testbed playground on a port-0 URL) with the seeded failing testseeds/fix-journey.open_session {test}, thenlocate,observe,close_session:
===== open_session {"test":"tests/journey.e2e.ts"} (30.8 s)
Test "an admin signs in, saves a profile, and creates a workspace" (tests/journey.e2e.ts:4) failed after 30.32s; 8 steps ran.
LOCATOR_NOT_FOUND: locator matched no nodes within getByLabel("Name")
At tests/journey.e2e.ts:14:3
12|
13| await app.open('/forms');
14| await screen.getByLabel('Name').fill('Grace Hopper');
| ^
15| await screen.getByLabel('Team').selectOption('Mobile');
16| await screen.getByLabel('Email notifications').check();
URL at failure: http://127.0.0.1:50536/forms
On screen, closest to what the locator asked for:
#n31 text="Full name"
#n32 textbox "Full name" purpose=username
Steps:
1. ✓ app.open /login (tests/journey.e2e.ts:6)
...
7. ✓ app.open /forms (tests/journey.e2e.ts:13)
8. ✗ locator.fill getByLabel("Name") LOCATOR_NOT_FOUND (tests/journey.e2e.ts:14)
The session is on the screen where the test stopped, shown below: observe and locate from here, and act out the test's next lines to check them too, then fix the test file. Its afterEach hooks and fixture teardown run at close_session. To run the fixed test from the start, close_session, then open_session {test: "tests/journey.e2e.ts"}.
Session 01a15e7a-... open on target "web" (platform web, engine web 0.15.2), headless; config .../e2e.config.ts.
...
Current screen (revision b3, path /forms, 45 nodes):
...
#n31 text="Full name"
#n32 textbox "Full name" purpose=username
===== call {"tool":"locate","args":{"label":"Full name"}} (0.0 s)
1 node matches getByLabel("Full name").
Use: screen.getByLabel("Full name")
===== call {"tool":"observe"} (0.1 s)
Current screen (revision b5, path /forms, 45 nodes):
...
- The same project with the fixed test (
seeds/fixed-journey) and anafterEachadded: the test passed in 427 ms and the session sat on/wizard. TheafterEachran only atclose_session, and a failing one showed asCleanup (afterEach): LOCATOR_NOT_FOUND: ...in its result. - Selection errors on the same server, before anything booted:
NO_TESTS: no test file matched nope; the config globs discovered tests/auth.setup.e2e.ts, tests/journey.e2e.ts, tests/many.e2e.ts; name a test file the config finds, with :line for one test in it
TEST_AMBIGUOUS: "many" selects 5 tests on target "web", and a session runs one; name it as file:line: tests/many.e2e.ts:3 "first", tests/many.e2e.ts:7 "second", tests/many.e2e.ts:12 "flow › step one", tests/many.e2e.ts:15 "flow › step two", tests/many.e2e.ts:20 "signed in"
SESSION_TEST_UNSUPPORTED: test "flow › step one" (tests/many.e2e.ts:12) is in a serial group, whose tests run in order on one shared attempt; a session runs a test alone, so run the group with npx e2e run
SESSION_TEST_UNSUPPORTED: test "signed in" (tests/many.e2e.ts:20) starts from session "admin", which a setup test saves first; a session does not run setup tests yet, so run it with npx e2e run
tests/integration/mcp-test-run.test.ts, over the built CLI and an MCP client: a pass on its final screen withafterEachat close, a failure with its line, frame, and nearest nodes and anobserveafter it, a rerun that sees an edited helper, a forgottenawaitthat fails the test and leaves the server serving, a late rejection reported at close and one after close as a warning, a body past its timeout that can't navigate the session, abeforeAllfailure reported once, a screenshot baseline that passes in the next session, and the selection errors. In-process: two sessions that collect and run side by side, an open cancelled while queued that runs none of its test, and a session's recording that replays in the next session and ine2e runwith no model call. The side-by-side, cancelled, and timed-out cases each fail when the fix they cover is removed fromdist.- The testbed's own
tests/forms.e2e.ts:9(adescribewith abeforeEach) passes in a session, andtests/forms.e2e.tsalone isTEST_AMBIGUOUSlisting its five tests. The testbed's forms, wizard, and smoke files still pass undere2e run.
A/B: a fresh agent fixing the seeded test
run-arm.mjs, Sonnet 5.5, the seeded test with three mistakes that surface one per run, each failing run waiting out the 30 s action timeout. Main baselines are the harness's own (main-fix-*) and PR B's main arms of the no-source variant (b-main-fixnosrc-*). Arms 1 to 4 ran before #1309 landed, with the new text as a step 6 of the writing workflow: f1-fix-1 and -2 with a first wording, the rest with two later ones that differ only in how hard the step pushes toward a session. Arms 5 ran the text as it sits now inside #1309's fix workflow, before the last wording fix on which line names a test.
| Run | Task | Wall min | e2e runs | Sessions | MCP calls | Tool calls | Model steps | Cost $ | Final exit |
|---|---|---|---|---|---|---|---|---|---|
| main-fix-1 | fix | 1.7 | 4 | 0 | 0 | 14 | 12 | 0.10 | 0 |
| main-fix-2 | fix | 1.1 | 2 | 0 | 0 | 14 | 11 | 0.06 | 0 |
| f1-fix-1 | fix | 1.5 | 4 | 0 | 0 | 11 | 7 | 0.10 | 0 |
| f1-fix-2 | fix | 1.6 | 4 | 0 | 0 | 11 | 7 | 0.08 | 0 |
| f1-fix-3 | fix | 1.5 | 2 | 1 | 6 | 15 | 11 | 0.13 | 0 |
| f1-fix-4 | fix | 2.1 | 3 | 1 | 7 | 16 | 12 | 0.11 | 0 |
| f1-fix-5 | fix | 1.1 | 3 | 0 | 0 | 13 | 9 | 0.10 | 0 |
| b-main-fixnosrc-1 | no source | 1.6 | 4 | 0 | 0 | 11 | 7 | 0.08 | 0 |
| b-main-fixnosrc-2 | no source | 1.5 | 4 | 0 | 0 | 10 | 6 | 0.08 | 0 |
| f1-fixnosrc-1 | no source | 1.5 | 4 | 0 | 0 | 10 | 6 | 0.08 | 0 |
| f1-fixnosrc-2 | no source | 2.1 | 3 | 1 | 7 | 15 | 10 | 0.10 | 0 |
| f1-fixnosrc-3 | no source | 1.5 | 4 | 0 | 0 | 10 | 6 | 0.08 | 0 |
| f1-fixnosrc-4 | no source | 1.5 | 4 | 0 | 0 | 10 | 7 | 0.08 | 0 |
| f1-fixnosrc-5 | no source | 1.5 | 4 | 1 | 1 | 11 | 8 | 0.10 | 0 |
- No speedup shows. On this small app Sonnet fixes each mistake from the terminal's nearest nodes and the trace page, and 6 of the 10 branch arms never opened a session.
main-fix-2's andf1-fix-5's low run counts came from reading the app's source outside the project, which the no-source task forbids. - The 3 arms that opened a session did what the feature is for. In
f1-fix-3, after one CLI run and the first fix, the agent opened{test: "tests/journey.e2e.ts"}. That landed on the wizard's wrong button, and the agent tappedNext, pickedPro, tappedNextandCreate workspace, and read the summary, all in the session. It fixed the last two mistakes together, and its one finale2e runpassed: 2 runs instead of 4.f1-fix-4andf1-fixnosrc-2did the same after a second CLI run, so the session paid the wizard's 30 s timeout a second time. f1-fixnosrc-5passedtest: "tests/journey.e2e.ts:23", the wizard line it wanted to look at. That isNO_TESTS(:23 names no test (declared at line 4)), and the agent went back to the CLI. The last commit says in the tool description, theNO_TESTShint, the skill, and the reference that the line is thetest(call's, and that a file holding one test needs none. Accepting a line inside a test would be friendlier, but it departs frome2e run's positional rules, so I left that for the maintainer.- The session pays the same action timeout as a run. What it saves is the failing rerun per remaining mistake, and the walk from the start over MCP the maintainer described. Neither shows much on a small app where the CLI output names the fix.
Review
Two fresh-context review rounds before opening. Fixed: a forgotten await that crashed the server, screenshot markers that outlived a session, concurrent collections that clashed, a cancelled open that still ran its hooks, a beforeAll failure reported twice, stale comments and docs (including docs/cache.mdx), a late rejection that could still crash the server, a timed-out body that kept driving the session, and a rejection after the verdict that flipped a passed outcome. Declined: extracting a shared body runner out of executeAttempt (below).
For the maintainer
standalone-test.tsrepeats the body part ofexecuteAttempt:beforeEach, the body race, steps not awaited, soft failures, settling, the teardown loop, and cache settlement. Sharing it means pulling a body and teardown runner out ofexecuteAttempt, with "stop before teardown" and "budget goes back to the session" as options. That is a refactor of the run path, so I left it for its own PR.- Tests that start from a setup test's
sessionare refused. That's the biggest gap for apps that sign in through a setup test. - Whether
open_session {test: "file:<line inside a test>"}should pick the test around that line. Agents pass the failing line;e2e rundoesn't accept it either.