docs(skills): write each test line while walking the flow over MCP

Merged
KA
21d ago
+181-79

The skill told the coding agent to learn the screens over e2e mcp first and write the test afterwards. Agents did exactly that: walk the whole flow, close the session, then write the file in one go from what they remember. Now the agent writes the test while it walks.

  • skills/e2e/SKILL.md, "Workflow: writing a test", step 2: with a connected e2e mcp server, create the test file before open_session, opening on the screen it covers, then add each step to the file as you take it. An act goal goes in before you walk its path. A screen line is the Test line: its action printed (#1317), copied as is. When the outcome shows, locate it and add the expect. Without a server, the old fallback stays (read the components, a --headed run). Step 3 now opens with "A step is an agent.act goal or screen lines" and keeps #1311's "start the test on the screen it covers" sentence.
  • skills/e2e/references/mcp.md, "Workflow", steps 1 to 3:
    1. Create the file, open_session, go to the screen the test opens on (navigate on the web, the shortest path on a device). Actions that only got you there, such as a sign-in a saved session replaces, stay out of the test when the test's opening reaches that screen directly.
    2. Walk one screen at a time and write each screen's steps before you leave it. An agent.act goal is written before its path, and the path's actions and their test lines stay out. A screen line is the action's Test line:. locate comes in only when the action printed No test line:.
    3. Each outcome gets its check before the next step: an expect on a locator tried with locate, or an agent.assert. test_lines is the catch-up for a line you missed. The step ends with an example walk.
  • Steps 4 and 5 are main's: #1309's "fix from the failure block and the trace page, go straight to a missing screen" and close_session. The "Workflow: fixing a failed test" section in SKILL.md is untouched.
  • Which kind of line a step becomes (my call, kept within the skill's existing rules):
    • agent.act is the default, for a path that varies. Write the goal in the words on screen, with its values in params, before you walk its path.
    • A screen line is for an exact value or interaction: a sign-in form in a setup test, a field that must receive one specific string (SKILL.md's list), or an interaction that is the point of the test (writing-tests' empty submit).
  • Take a screen's actions, then add the lines they printed in one edit, so writing as you walk costs a turn per screen, not a turn per line.
  • The e2e mcp server instructions (packages/e2e/src/mcp/server.ts:55) said "Write deterministic tests (tests/.e2e.ts) from what you saw", the opposite of the new skill, and every MCP client reads them. They now say "Write the test (tests/.e2e.ts) while you walk the flow, adding each step to the file as you take it". #1317's sentence before it already says each action answers with its test line. e2e mcp --help lists test_lines and says "Each action prints the test line that repeats it, so the agent writes a test while it walks the real app." The opening sentence of docs/reference/mcp.mdx changed to "while they write tests".
  • Docs: docs/coding-agents.mdx ("Let the agent use the app", "How an agent writes and runs a test") and docs/mcp-server.mdx (the "Write a test" row, the example prompt, #1317's "How it writes a test line", and the lede of "How it checks a locator", which is now about lines no action printed).
  • type_secret's row in mcp.md's catalog now says it prints a test line too, since a sign-in form is one of the screen cases.
  • Not touched: failure reading, reruns, the cache, and session closing. One clause in an mcp.md Rules bullet changed, "the test is what you write afterwards" -> "the test is the file you write as you walk", because it contradicted the new workflow.

The walk from mcp.md:

write   tests/members.e2e.ts: imports, test(...), app.open('/settings/members')
call    open_session, then navigate {url: "/settings/members"}   its line is the app.open already written
edit    + await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } });
call    tap, type, select, tap                   the goal's path; its test lines stay out
call    locate {role: "status"}                  1 node matches, text "Invite sent"
edit    + await expect(screen.getByRole('status')).toContainText('Invite sent');
call    type {target: "n40", value: "ada"}       prints the test line below
        Test line: await screen.getByRole("searchbox", "Search members").fill("ada");
edit    + that line, as printed
call    locate {role: "listitem"}                1 node matches
edit    + await expect(screen.getByRole('listitem')).toHaveCount(1);

A/B: does a fresh agent follow it?

These runs predate the rebase onto #1317. The branch wording then had the agent locate every control before writing its line; it now copies the printed Test line: and calls locate only for assertions and controls with no line. The order of events (file first, write per screen) is what the A/B measured, and that part of the wording is unchanged. The locate counts below would drop with the current wording. I haven't rerun the harness.

The harness ran a fresh OpenCode agent (anthropic/claude-sonnet-5-5) on the "author one journey" task: sign in, a profile form, the workspace wizard on the testbed playground, with the e2e MCP server and the skill under test. The scratch config has no model, so every arm wrote screen lines only.

"Writes" means file writes before open_session / during the walk / after the last action.

RunSkillWall mine2e runsSessionsMCP callslocateWritesTool callsModel stepsCost $Final exit
main-author-1main0.8111910/0/126170.160
main-author-2main0.9111700/0/123140.140
a-main-author-1main, rerun today0.8111700/0/122150.180
a-main-author-2main, rerun today0.6111700/0/125130.190
a-author-7this branch1.41130131/6/144200.250
a-author-8this branch1.31131141/6/146230.270
  • Order of events. All four main arms walk the whole flow, then write the file once at the end (events.jsonl: 15 to 17 e2e_calls, then one write just before or after e2e_close_session). Both branch arms write the skeleton before open_session. Then, on each screen, they locate its controls in one turn and add the lines in the same turn as the actions. They write 8 times in all, and the last edit adds the wizard's expect before close_session.
  • Locators. The branch arms tried 13 to 14 locators with locate before writing them. Main tried 0 or 1 and wrote names from observe.
  • Cost. On this small app, main's tests also pass on the first run, so the A/B shows no gain in correctness. It shows the price: 3 to 10 more model steps, $0.06 to $0.09 more, and 0.5 to 0.8 more minutes than today's main reruns. Most of the extra MCP calls were those locates, which the printed test lines now replace. The benefit is meant for long flows and apps whose rendered names differ from what an agent would guess, and this harness can't show it. n=2 per arm, so read the numbers as a sanity check.
  • Not exercised: the agent.act half of the walk (goal first, then walk its path), since the harness config has no model.
Earlier wordings of this PR, same task
RunWordingWall minMCP callslocateWritesModel stepsCost $
a-author-1first draft, "write the step, then do it"0.91920/0/1170.25
a-author-2first draft0.728110/0/1170.26
a-author-3+ "do not walk first", file before open_session, example0.931141/3/1180.24
a-author-4same1.030131/5/1180.24
a-author-5+ server instructions1.033151/7/1230.28
a-author-6same1.133151/6/1230.30

The first draft didn't change the order: both arms still walked first and wrote at the end. What changed the order was the explicit "do not walk the whole flow first", creating the file before open_session, and the example walk.

Verified

Ran it locally: yes, after rebasing onto main at 72abdddd (with #1309, #1310, #1311, #1317). Web only.

After merging main again at cfe76ba3 (with #1313 and #1314): skills/e2e/references/mcp.md conflicted twice. The walk (steps 1 to 3 and the example) is this PR's; step 4 is #1314's "run only that test" by file:line; step 5 is #1313's "leave the session open". In Rules, "the test is the file you write as you walk" leads into #1313's rules for running beside a session, which replace the old reuseExisting line. SKILL.md merged without conflicts and reads as one workflow. Re-ran pnpm check, pnpm plugins:check, and skill.test.ts on the merge.

  • Built package: node packages/e2e/dist/cli/bin.js guide and guide mcp print the merged steps, with #1309's fixing workflow and mcp.md step 4 intact. e2e mcp --help:

    call runs the session's tools (observe, tap, type, press, select, scroll,
    navigate, type_secret, locate, test_lines, screenshot, and the project's own),
    tools describes them, close_session ends it. Each action prints the test line
    that repeats it, so the agent writes a test while it walks the real app.
    
  • MCP server, the walk the skill describes, on the testbed's /todos (a port-0 copy of apps/testbed/e2e.config.ts, not committed). A real MCP client against the built e2e mcp. The instructions it gets end with "Write the test (tests/*.e2e.ts) while you walk the flow, adding each step to the file as you take it, and run it with the CLI". The actions print:

    Navigated to /todos.
    Test line: await app.open("/todos");
    Typed into #n6028.
    Test line: await screen.getByRole("textbox", "New todo").fill("Buy milk");
    Tapped #n6029.
    Test line: await screen.getByRole("button", "Add").tap();
    1 node matches getByRole("listitem").
    Use: screen.getByRole("listitem")
    

    Those three lines copied as printed, plus an expect on the located listitem, pass e2e run on the first try:

     ✓ |web| tests/pr1312-walk.e2e.ts (1 test) 224ms
       ✓ adds a todo 224ms
    
  • Mobile: the device half of step 1 ("the shortest path from the launch screen") follows #1309's wording and the absence of navigate in @e2e-dev/mobile. Not run on a simulator or emulator.

  • The A/B above ran before the rebase (see its note).

  • Docs: rendered with mint dev after the rebase. Before -> after against main: mcp-server.mdx 137 -> 138 lines, 587 -> 607 prose words; coding-agents.mdx 264 -> 266 lines, 1055 -> 1076 words. Shortening pass on the touched sections: "and adds the assertions" left #1317's "How it writes a test line", since the next section covers assertions.

docs/mcp-server.mdx tabledocs/mcp-server.mdx lines and locatorsdocs/coding-agents.mdx
mcp server tablemcp server linescoding agents

Review

Fresh-context review before the rebase, findings fixed:

  • open_session opens the app's base URL, not the path app.open names, so the walk could start on another screen than the test. Step 1 and the example now navigate to the path first.
  • mcp.md's screen list was wider than SKILL.md's, and the first example filled an invite form with screen, which agent.md writes as agent.act with params. The lists now match. The example's invite is an act goal with params, and a screen line comes from the exact string the count depends on.
  • SKILL.md step 3 told agents without an MCP server to locate. The walk and locate now live only under step 2's MCP branch.
  • The mcp.md opening line said the testing agent writes tests. Reworded.
  • "Every locator has matched once" over-claimed, since locate can't try a .filter() chain. It now reads "every locator you copied from a Test line: or a locate result", and says the first run checks the rest.
  • Each docs page said "writes while it walks" twice. Now it's once per page, and coding-agents.mdx says "with e2e mcp".

Fresh-context review after the rebase, fixed:

  • The "Write a test" row said the agent copies a line from each action, but an act goal's path stays out. It now reads "Writes the test as it walks the flow, from checked test lines".
  • locate was explained in both step 2 and step 3; it now lives in step 3, with the outcome it checks. Step 3 no longer says to locate an agent.assert.
  • "Add each step as you take it" against "add a screen's lines in one edit": step 2 now says to write each screen's steps before you leave it.
  • On a device, or a screen the test can't open directly, the steps that reach it are the test's first steps, not left out.
  • The example's edits now carry await, like the copied line, and say that navigate's line is the app.open already written.
  • "checks each locator" (mcp.md opening, coding-agents.mdx) became "checks its locators", for the same .filter() reason.

Left for a follow-up (outside this PR's files, or #1317's code):

  • test_lines' own description and footer (packages/e2e/src/mcp/test-lines.ts:191, :209) say "Paste it into a test", which reads as pasting the whole draft, act goals' paths included.
  • skills/e2e/references/bug-bash.md (regression tests), skills/migrate-to-e2e/SKILL.md:167, and the generated verification-skill example still say to locate each locator before writing it. They could point at the printed test lines instead.