docs(skills): write each test line while walking the flow over MCP
The skill told the coding agent to learn the screens over e2e mcp first and write the test afterwards. Agents did exactly that: walk the whole flow, close the session, then write the file in one go from what they remember. Now the agent writes the test while it walks.
skills/e2e/SKILL.md, "Workflow: writing a test", step 2: with a connectede2e mcpserver, create the test file beforeopen_session, opening on the screen it covers, then add each step to the file as you take it. Anactgoal goes in before you walk its path. Ascreenline is theTest line:its action printed (#1317), copied as is. When the outcome shows,locateit and add theexpect. Without a server, the old fallback stays (read the components, a--headedrun). Step 3 now opens with "A step is anagent.actgoal orscreenlines" and keeps #1311's "start the test on the screen it covers" sentence.skills/e2e/references/mcp.md, "Workflow", steps 1 to 3:- Create the file,
open_session, go to the screen the test opens on (navigateon the web, the shortest path on a device). Actions that only got you there, such as a sign-in a saved session replaces, stay out of the test when the test's opening reaches that screen directly. - Walk one screen at a time and write each screen's steps before you leave it. An
agent.actgoal is written before its path, and the path's actions and their test lines stay out. Ascreenline is the action'sTest line:.locatecomes in only when the action printedNo test line:. - Each outcome gets its check before the next step: an
expecton a locator tried withlocate, or anagent.assert.test_linesis the catch-up for a line you missed. The step ends with an example walk.
- Create the file,
- Steps 4 and 5 are main's: #1309's "fix from the failure block and the trace page, go straight to a missing screen" and
close_session. The "Workflow: fixing a failed test" section inSKILL.mdis untouched. - Which kind of line a step becomes (my call, kept within the skill's existing rules):
agent.actis the default, for a path that varies. Write the goal in the words on screen, with its values inparams, before you walk its path.- A
screenline is for an exact value or interaction: a sign-in form in a setup test, a field that must receive one specific string (SKILL.md's list), or an interaction that is the point of the test (writing-tests' empty submit).
- Take a screen's actions, then add the lines they printed in one edit, so writing as you walk costs a turn per screen, not a turn per line.
- The
e2e mcpserver instructions (packages/e2e/src/mcp/server.ts:55) said "Write deterministic tests (tests/.e2e.ts) from what you saw", the opposite of the new skill, and every MCP client reads them. They now say "Write the test (tests/.e2e.ts) while you walk the flow, adding each step to the file as you take it". #1317's sentence before it already says each action answers with its test line.e2e mcp --helpliststest_linesand says "Each action prints the test line that repeats it, so the agent writes a test while it walks the real app." The opening sentence ofdocs/reference/mcp.mdxchanged to "while they write tests". - Docs:
docs/coding-agents.mdx("Let the agent use the app", "How an agent writes and runs a test") anddocs/mcp-server.mdx(the "Write a test" row, the example prompt, #1317's "How it writes a test line", and the lede of "How it checks a locator", which is now about lines no action printed). type_secret's row in mcp.md's catalog now says it prints a test line too, since a sign-in form is one of thescreencases.- Not touched: failure reading, reruns, the cache, and session closing. One clause in an mcp.md Rules bullet changed, "the test is what you write afterwards" -> "the test is the file you write as you walk", because it contradicted the new workflow.
The walk from mcp.md:
write tests/members.e2e.ts: imports, test(...), app.open('/settings/members')
call open_session, then navigate {url: "/settings/members"} its line is the app.open already written
edit + await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } });
call tap, type, select, tap the goal's path; its test lines stay out
call locate {role: "status"} 1 node matches, text "Invite sent"
edit + await expect(screen.getByRole('status')).toContainText('Invite sent');
call type {target: "n40", value: "ada"} prints the test line below
Test line: await screen.getByRole("searchbox", "Search members").fill("ada");
edit + that line, as printed
call locate {role: "listitem"} 1 node matches
edit + await expect(screen.getByRole('listitem')).toHaveCount(1);
A/B: does a fresh agent follow it?
These runs predate the rebase onto #1317. The branch wording then had the agent locate every control before writing its line; it now copies the printed Test line: and calls locate only for assertions and controls with no line. The order of events (file first, write per screen) is what the A/B measured, and that part of the wording is unchanged. The locate counts below would drop with the current wording. I haven't rerun the harness.
The harness ran a fresh OpenCode agent (anthropic/claude-sonnet-5-5) on the "author one journey" task: sign in, a profile form, the workspace wizard on the testbed playground, with the e2e MCP server and the skill under test. The scratch config has no model, so every arm wrote screen lines only.
"Writes" means file writes before open_session / during the walk / after the last action.
| Run | Skill | Wall min | e2e runs | Sessions | MCP calls | locate | Writes | Tool calls | Model steps | Cost $ | Final exit |
|---|---|---|---|---|---|---|---|---|---|---|---|
| main-author-1 | main | 0.8 | 1 | 1 | 19 | 1 | 0/0/1 | 26 | 17 | 0.16 | 0 |
| main-author-2 | main | 0.9 | 1 | 1 | 17 | 0 | 0/0/1 | 23 | 14 | 0.14 | 0 |
| a-main-author-1 | main, rerun today | 0.8 | 1 | 1 | 17 | 0 | 0/0/1 | 22 | 15 | 0.18 | 0 |
| a-main-author-2 | main, rerun today | 0.6 | 1 | 1 | 17 | 0 | 0/0/1 | 25 | 13 | 0.19 | 0 |
| a-author-7 | this branch | 1.4 | 1 | 1 | 30 | 13 | 1/6/1 | 44 | 20 | 0.25 | 0 |
| a-author-8 | this branch | 1.3 | 1 | 1 | 31 | 14 | 1/6/1 | 46 | 23 | 0.27 | 0 |
- Order of events. All four main arms walk the whole flow, then write the file once at the end (
events.jsonl: 15 to 17e2e_calls, then onewritejust before or aftere2e_close_session). Both branch arms write the skeleton beforeopen_session. Then, on each screen, theylocateits controls in one turn and add the lines in the same turn as the actions. They write 8 times in all, and the last edit adds the wizard'sexpectbeforeclose_session. - Locators. The branch arms tried 13 to 14 locators with
locatebefore writing them. Main tried 0 or 1 and wrote names fromobserve. - Cost. On this small app, main's tests also pass on the first run, so the A/B shows no gain in correctness. It shows the price: 3 to 10 more model steps, $0.06 to $0.09 more, and 0.5 to 0.8 more minutes than today's main reruns. Most of the extra MCP calls were those
locates, which the printed test lines now replace. The benefit is meant for long flows and apps whose rendered names differ from what an agent would guess, and this harness can't show it. n=2 per arm, so read the numbers as a sanity check. - Not exercised: the
agent.acthalf of the walk (goal first, then walk its path), since the harness config has no model.
Earlier wordings of this PR, same task
| Run | Wording | Wall min | MCP calls | locate | Writes | Model steps | Cost $ |
|---|---|---|---|---|---|---|---|
| a-author-1 | first draft, "write the step, then do it" | 0.9 | 19 | 2 | 0/0/1 | 17 | 0.25 |
| a-author-2 | first draft | 0.7 | 28 | 11 | 0/0/1 | 17 | 0.26 |
| a-author-3 | + "do not walk first", file before open_session, example | 0.9 | 31 | 14 | 1/3/1 | 18 | 0.24 |
| a-author-4 | same | 1.0 | 30 | 13 | 1/5/1 | 18 | 0.24 |
| a-author-5 | + server instructions | 1.0 | 33 | 15 | 1/7/1 | 23 | 0.28 |
| a-author-6 | same | 1.1 | 33 | 15 | 1/6/1 | 23 | 0.30 |
The first draft didn't change the order: both arms still walked first and wrote at the end. What changed the order was the explicit "do not walk the whole flow first", creating the file before open_session, and the example walk.
Verified
Ran it locally: yes, after rebasing onto main at 72abdddd (with #1309, #1310, #1311, #1317). Web only.
After merging main again at cfe76ba3 (with #1313 and #1314): skills/e2e/references/mcp.md conflicted twice. The walk (steps 1 to 3 and the example) is this PR's; step 4 is #1314's "run only that test" by file:line; step 5 is #1313's "leave the session open". In Rules, "the test is the file you write as you walk" leads into #1313's rules for running beside a session, which replace the old reuseExisting line. SKILL.md merged without conflicts and reads as one workflow. Re-ran pnpm check, pnpm plugins:check, and skill.test.ts on the merge.
-
Built package:
node packages/e2e/dist/cli/bin.js guideandguide mcpprint the merged steps, with #1309's fixing workflow and mcp.md step 4 intact.e2e mcp --help:call runs the session's tools (observe, tap, type, press, select, scroll, navigate, type_secret, locate, test_lines, screenshot, and the project's own), tools describes them, close_session ends it. Each action prints the test line that repeats it, so the agent writes a test while it walks the real app. -
MCP server, the walk the skill describes, on the testbed's
/todos(a port-0 copy ofapps/testbed/e2e.config.ts, not committed). A real MCP client against the builte2e mcp. The instructions it gets end with "Write the test (tests/*.e2e.ts) while you walk the flow, adding each step to the file as you take it, and run it with the CLI". The actions print:Navigated to /todos. Test line: await app.open("/todos"); Typed into #n6028. Test line: await screen.getByRole("textbox", "New todo").fill("Buy milk"); Tapped #n6029. Test line: await screen.getByRole("button", "Add").tap(); 1 node matches getByRole("listitem"). Use: screen.getByRole("listitem")Those three lines copied as printed, plus an
expecton the locatedlistitem, passe2e runon the first try:✓ |web| tests/pr1312-walk.e2e.ts (1 test) 224ms ✓ adds a todo 224ms -
Mobile: the device half of step 1 ("the shortest path from the launch screen") follows #1309's wording and the absence of
navigatein@e2e-dev/mobile. Not run on a simulator or emulator. -
The A/B above ran before the rebase (see its note).
-
Docs: rendered with
mint devafter the rebase. Before -> after against main:mcp-server.mdx137 -> 138 lines, 587 -> 607 prose words;coding-agents.mdx264 -> 266 lines, 1055 -> 1076 words. Shortening pass on the touched sections: "and adds the assertions" left #1317's "How it writes a test line", since the next section covers assertions.
docs/mcp-server.mdx table | docs/mcp-server.mdx lines and locators | docs/coding-agents.mdx |
|---|---|---|
Review
Fresh-context review before the rebase, findings fixed:
open_sessionopens the app's base URL, not the pathapp.opennames, so the walk could start on another screen than the test. Step 1 and the example nownavigateto the path first.- mcp.md's
screenlist was wider than SKILL.md's, and the first example filled an invite form withscreen, whichagent.mdwrites asagent.actwith params. The lists now match. The example's invite is anactgoal withparams, and ascreenline comes from the exact string the count depends on. - SKILL.md step 3 told agents without an MCP server to
locate. The walk andlocatenow live only under step 2's MCP branch. - The mcp.md opening line said the testing agent writes tests. Reworded.
- "Every locator has matched once" over-claimed, since
locatecan't try a.filter()chain. It now reads "every locator you copied from aTest line:or alocateresult", and says the first run checks the rest. - Each docs page said "writes while it walks" twice. Now it's once per page, and
coding-agents.mdxsays "withe2e mcp".
Fresh-context review after the rebase, fixed:
- The "Write a test" row said the agent copies a line from each action, but an
actgoal's path stays out. It now reads "Writes the test as it walks the flow, from checked test lines". locatewas explained in both step 2 and step 3; it now lives in step 3, with the outcome it checks. Step 3 no longer says tolocateanagent.assert.- "Add each step as you take it" against "add a screen's lines in one edit": step 2 now says to write each screen's steps before you leave it.
- On a device, or a screen the test can't open directly, the steps that reach it are the test's first steps, not left out.
- The example's edits now carry
await, like the copied line, and say thatnavigate's line is theapp.openalready written. - "checks each locator" (mcp.md opening,
coding-agents.mdx) became "checks its locators", for the same.filter()reason.
Left for a follow-up (outside this PR's files, or #1317's code):
test_lines' own description and footer (packages/e2e/src/mcp/test-lines.ts:191,:209) say "Paste it into a test", which reads as pasting the whole draft, act goals' paths included.skills/e2e/references/bug-bash.md(regression tests),skills/migrate-to-e2e/SKILL.md:167, and the generated verification-skill example still say tolocateeach locator before writing it. They could point at the printed test lines instead.