PR #1313 - tester-army/e2e

Today skills/e2e/references/mcp.md tells the agent to close the MCP session before every e2e run unless the app command sets reuseExisting. So after a red run, the agent opens a new session to see the failing screen. Since #1309 it goes straight to that screen instead of walking the flow, but still in a new session. This change keeps the session open between runs on the web, parked on the screen under test, so the live look happens in the session the agent already has.

references/mcp.md Rules: what a run beside an open session does, case by case, from the target's app. Each case was checked on the built CLI (see Verified):
- no command: both use that app
- port 0: each starts its own copy, except a dev server that allows one copy per directory (next dev in Next.js 16 fails the run with APP_UNREACHABLE)
- fixed port with reuseExisting: true: the run reuses the session's app, and whichever started the app stops it. The session closing during a run (close_session, 30 idle minutes, the 4-hour limit; a run is not session activity) stops the app under the run, so the skill says to observe once before a long run
- fixed port without it: APP_ALREADY_RUNNING
Config edits: the agent offers reuseExisting: true to the user and never adds it unasked (CI ignores it, and it is not part of the replay cache key). Until the user agrees, it closes the session before each run. I did not suggest switching a fixed port to port 0: it changes app.identity, which keys every recording, and it needs {port} support in the app's command.
CI set (many agent sandboxes): reuseExisting is ignored, so the agent prefixes the run with CI= or closes the session. Unsetting CI also drops the other CI defaults; the skill says so. bug-bash.md already gives the same advice.
Mobile: close the session before each run. A session stays open only on a target whose device no run uses, with --target on each run; adding a target is a config edit, so the agent asks.
references/mcp.md Workflow step 5, right after #1309's step 4 (fix from the failure block and trace page, go straight to a missing screen): leave the session open through those runs, so step 4's live look needs no new session, and close_session once the test passes. observe does not reload, so the agent navigates even to the screen the session is on when the run or a fix may have changed it. A live look's actions print #1317's Test line: too, and test_lines lists them after the first walk, so the step says to copy a fixed line from its action.
SKILL.md Rules: one line pointing at the above. In the A/B below, 14 of 18 agents never opened references/mcp.md, but every one loaded SKILL.md.
references/debugging.md: the APP_ALREADY_RUNNING row names an open session's app as a cause and says to close_session first. Otherwise the agent stops the process only if it started it, and offers reuseExisting instead of setting it.
docs/mcp-server.mdx "Run tests next to a session": the same cases as a table. After a failed run, the agent goes to the failing screen in that session instead of opening a new one. docs/web.mdx: the reuseExisting row links there.

Workflow steps 1 to 4 come from #1317 (test lines while walking) and #1309 (fixing from the trace page), unchanged here. SKILL.md's "Workflow: fixing a failed test" (#1309) already says "open or reuse a session" and sends the agent to topic mcp for when a session must close before a rerun; this PR's Rules are that answer.

Verified

Ran it locally: yes

After the rebase onto main (#1309, #1317, #1310, #1311). Only Markdown conflicted, so I did not rerun the side-by-side matrix below. I re-checked the one new claim, about test lines, on the built MCP server: the same harness project with a port-0 app, a session parked on /forms, e2e run tests/journey.e2e.ts beside it, then a live look in the same session:

walk: navigate /forms -> Test line: await app.open("/forms");
walk: type Full name -> Test line: await screen.getByRole("textbox", "Full name").fill("Grace Hopper");
e2e run beside the session: exit 0
  |  ✓ target "web" command ready 87ms
  |       Tests  1 passed (1)
live look: navigate /forms -> Test line: await app.open("/forms");
live look: tap Save profile -> Test line: await screen.getByRole("button", "Save profile").tap();
test_lines:
  await app.open();
  await app.open("/forms");
  await screen.getByRole("textbox", "Full name").fill("Grace Hopper");
  await app.open("/forms");
  await screen.getByRole("button", "Save profile").tap();

npx e2e guide mcp from the built package prints the merged steps 4 and 5.

The built CLI, a session and a run side by side (before the rebase). I used a harness project (the testbed playground app, node server.mjs) and drove e2e mcp with the MCP SDK client. The script opens a session, parks it on /forms, runs e2e run tests/journey.e2e.ts --retries 0, then calls observe in the session again:

Target's `app`	`e2e run` beside the session	Session after the run
fixed port, no `reuseExisting`	exit 3, `APP_ALREADY_RUNNING ... stop that process or set reuseExisting: true`	still on `/forms`
fixed port, `reuseExisting: true`	exit 0, `target "web" command reused 5ms`	still on `/forms`; `close_session` then stops the app
port 0 (`{port}` in `env`)	exit 0, `target "web" command ready 86ms` (its own app; the session's is on :63134)	still on `/forms`; `screenshot` and `start_recording` still work after the run cleared `.e2e/results`
`reuseExisting`, run with `CI=1`	exit 3, `reuseExisting is ignored in CI, starting the command` then `APP_ALREADY_RUNNING`	still on `/forms`
`reuseExisting`, run with `CI=`	exit 0, `command reused 7ms`, no `CI` in the run header	still on `/forms`
port 0, run with `CI=1`	exit 0, `command ready 88ms`	still on `/forms`
port 0, Next.js 16.4 `next dev --port {port}`	exit 3, `APP_UNREACHABLE ... exited with code 1`, log: `Another next dev server is already running`	still on `/`

Who owns a reused app (reuseExisting: true, a test that sleeps 8 s between two pages):

# session first, closed 3 s into the run
close_session during the run -> app at :47391 does not answer
e2e run exit 3: APP_UNREACHABLE: nothing answered at http://127.0.0.1:47391/wizard (ERR_CONNECTION_REFUSED)

# run first, session opened 3 s into it
e2e mcp: [info] app: target "web" command: reusing the process already serving http://127.0.0.1:47391/
e2e run exit 0
navigate /wizard after the run failed: APP_UNREACHABLE: ... net::ERR_CONNECTION_REFUSED

Not verified: mobile. I had no free simulator for a session and a run at once. The mobile text keeps what docs/reference/mcp.mdx and setup.md already say about one device per session.

A fresh agent (OpenCode, anthropic/claude-sonnet-5-5) with main's skill and with this branch's (before the rebase, so neither arm had #1309's or #1317's text), n=2 to 4 per arm, in a scratch testbed project with the app on port 0. The fix-mcp task is the seeded three-mistake fix-journey test plus "Use the e2e MCP server to look at the app", added so that sessions come into play. In the plain fix-journey task no agent opens a session on either skill. Arms marked * ran before the review fixes and the SKILL.md line; the harness outcome for a port-0 app is the same under both texts.

Run	Wall min	e2e runs	Sessions	MCP calls	Tool calls	Model steps	Cost $	Session open through the last run
main-author-1	0.8	1	1	19	26	17	0.16	no, closed before the run
main-author-2	0.9	1	1	17	23	14	0.14	no, closed before the run
c-author-1 *	0.7	1	1	17	23	14	0.18	yes
c-author-2 *	0.7	1	1	17	25	16	0.20	yes
c-author-3	0.8	1	1	18	26	16	0.21	no, closed before the run
c-author-4	0.8	1	1	18	24	15	0.20	yes
c-main-fixmcp-1	1.2	2	1	17	24	13	0.14	no, closed before the last run
c-main-fixmcp-2	1.7	3	1	9	17	12	0.11	no, closed before the last run
c-fixmcp-1 *	1.6	2	1	13	20	13	0.12	no, closed before the last run
c-fixmcp-2 *	1.7	3	1	9	17	12	0.12	yes
c-fixmcp-3	1.0	2	1	10	17	9	0.11	yes
c-fixmcp-4	1.1	2	1	13	20	10	0.11	no, closed before the last run
main-fix-1	1.7	4	0	0	14	12	0.10	no session
main-fix-2	1.1	2	0	0	14	11	0.06	no session
c-fix-1 *	2.0	4	0	0	10	9	0.09	no session
c-fix-2 *	1.2	3	0	0	16	12	0.15	no session
c-main-fixedport-fixmcp-1	0.8	2	1	17	25	16	0.14	no, closed after `APP_ALREADY_RUNNING`
c-fixedport-fixmcp-1	0.8	2	1	17	24	13	0.13	no, closed after `APP_ALREADY_RUNNING`

The behavior moved. With this branch, 5 of 8 arms kept the session open through the last run and closed it after the pass. With main, 0 of 4 did.
3 of the 4 author arms on this branch read references/mcp.md, and c-author-3 closed the session before its run even after reading it. c-author-1 kept it open without reading it. Model variance is large at this n.
No speedup shows, and none should be claimed. On this small app every agent opened exactly one session and fixed all three mistakes from one walk, so there was never a second walk to save. Wall time, steps, and cost are within noise of main.
Fixed port without reuseExisting (c-fixedport-fixmcp-1 on this branch, c-main-fixedport-fixmcp-1 on main, via a copy of the harness with a fixed-port option): both agents hit APP_ALREADY_RUNNING on their first run, closed the session, and passed on the next. Neither edited e2e.config.ts. The branch agent never opened references/mcp.md, so it did not offer reuseExisting either.

Docs page, rendered with mint dev before the rebase. The rebase rewords the intro's last sentence only; mint validate and mint broken-links pass on it:

For the maintainer

The MCP server tells the agent to close the session too: its instructions say "close_session when done" (packages/e2e/src/mcp/server.ts:53), and the open_session text ends with "Call close_session when you are done." (packages/e2e/src/mcp/session.ts:442). Agents read that text on every session; most never open references/mcp.md. #1317 rewrote both strings for test lines and kept the close advice. Aligning them is a code change outside this docs PR, so I left it out.
bug-bash.md step 1 recommends port 0 for parallel explorers. That has the same next dev gap and is out of this PR's scope.
docs/debugging.mdx has the same APP_ALREADY_RUNNING row for people and does not name an open session; I left that page alone.
The shared A/B harness's --reuse-existing flag sets reuseExisting on a port-0 URL, which the config rejects with INVALID_CONFIG.

#1313 docs(skills): keep the MCP session open between test runs

Verified

For the maintainer

docs(skills): keep the MCP session open between test runs