PR #1317 - tester-army/e2e

Every grammar action in an e2e mcp session now answers with the test line that repeats it, so a coding agent can write the test while it walks the flow, without a locate call per control.

Tapped #n6031.
Test line: await screen.getByRole("button", "Sign in").tap();

Screen changes since revision b4 ...

Where the locator comes from. The target node as the newest observation listed it, read before the action runs, since the action may remove it. The session's context asks every observation for its tree and drops it again unless the caller asked (packages/e2e/src/mcp/test-lines.ts, watch).
Which locator. Role and name, label, placeholder, text, then the test id read off the live node (the observation tree carries none), in the skill's order. Each is built with locateQuery and tried with LocatorEngine.resolveNow, as locate does. The first that matches the target and nothing else wins.
Twins. A locator that also matches twins is narrowed to { visible: true }, then scoped under a container (getByRole("listitem").filter({ hasText: "Offer B" })), then pinned with .nth(i) plus a // comment. A position is printed only when the live twins are the ones the observation listed, each in place or all shifted by one offset, or in their boxes in another order (light DOM before shadow roots). Otherwise the result says No test line: and why.
Never a line that fails. No locator reaches the target alone, a failed action, typing into the focused field, press with times, scroll, scroll_to a text, dismiss_keyboard, and a type the keyboard took all print a reason instead. Point tools print nothing. At most 10 locators are tried per target, since each one is a full snapshot on a device.
Verbs. tap(), doubleTap(), longPress(), secondaryTap(), hover() (with modifiers), fill(value) (grammar type with a target replaces the value, so fill, not pressSequentially), press(key), selectOption(label), check()/uncheck(), dragTo(locator), setInputFiles(...), scrollIntoView(), app.open(path) (a URL on the app's origin becomes its path, so the line survives the next run's port), app.back(), and app.open() for the session's own opening.
Secrets. type_secret writes credentials.user(name).password or secrets.get(name). A typed value equal to a credential's username becomes credentials.user(name).username. Results and test_lines pass the attempt's secret ledger like every session result.
test_lines. A read-only catalog tool that returns every line so far, in order, as a draft test body, with a // comment where an action had none and a note on which of credentials/secrets to import.
MCP only. createSessionCatalog wraps the grammar tools; the testing agent's loop never sees a line. test_lines joins the reserved tool names (a project tool can no longer take it) and telemetry now derives its session tool set from that list, so test_lines calls aren't folded into other.

flowchart LR
  A["call tap #n42"] --> B["node #n42 from the newest observation tree"]
  B --> C{"role+name, label, placeholder, text, test id: resolveNow matches only the target?"}
  C -- yes --> E["run the action"]
  C -- twins --> D{"visible: true, container scope, certain position?"}
  D -- yes --> E
  D -- no --> F["reason"]
  F --> E
  E --> G{"action failed?"}
  G -- no --> H["Test line / No test line after the lead; recorded for test_lines"]
  G -- yes --> I["result as before, nothing recorded"]

Docs: docs/reference/mcp.mdx (catalog row, test_lines entry with the verb table), docs/mcp-server.mdx ("How it writes a test line"), docs/tools.mdx and docs/reference/errors.mdx (reserved name). Skill: skills/e2e/references/mcp.md (tools table, workflow step 1 says to copy each line, step 2 keeps locate for what no action printed) and one clause in SKILL.md.

Decisions I made that the maintainer may want to revisit:

Point tools print no line, not screen.tapAt(...): screenshot coordinates break on the next viewport, and a listed control should be written as a locator.
.nth(i) is printed (with a comment) rather than no line when only a certain position tells twins apart.
scroll and scroll_to a text print a reason, not screen.scrollUntilVisible(...), since the session can't check that line before the scroll.

Verified

Ran it locally: yes

The built MCP server on the testbed app (an A/B harness --setup-only project, driven by a small MCP SDK client). The author-journey flow, one action per call:

>>> type {"target":"n6028","value":"admin"}
Typed into #n6028.
Test line: await screen.getByRole("textbox", "Username").fill(credentials.user("admin").username);

>>> type_secret {"target":"n6030","name":"admin.password"}
Filled secret "admin.password" into #n6030; its value is masked in every observation.
Test line: await screen.getByRole("textbox", "Password").fill(credentials.user("admin").password);

>>> select {"target":"n36037","value":"Mobile"}
Selected "Mobile" in #n36037.
Test line: await screen.getByRole("combobox", "Team").selectOption("Mobile");

>>> check {"target":"n36044"}
Checked #n36044.
Test line: await screen.getByRole("checkbox", "Email notifications").check();

test_lines at the end:

Draft test body from this session's actions, in order (16 lines):

await app.open();
await app.open("/login");
await screen.getByRole("textbox", "Username").fill(credentials.user("admin").username);
await screen.getByRole("textbox", "Password").fill(credentials.user("admin").password);
await screen.getByRole("button", "Sign in").tap();
await app.open("/forms");
await screen.getByRole("textbox", "Full name").fill("Grace Hopper");
await screen.getByRole("combobox", "Team").selectOption("Mobile");
await screen.getByRole("checkbox", "Email notifications").check();
await screen.getByRole("button", "Save profile").tap();
await app.open("/wizard");
await screen.getByRole("textbox", "Workspace name").fill("Atlas");
await screen.getByRole("button", "Next").tap();
await screen.getByRole("combobox", "Plan").selectOption("Pro");
await screen.getByRole("button", "Next").tap();
await screen.getByRole("button", "Create workspace").tap();

Paste it into a test body, async ({ app, screen }) => { ... }, and add an expect after each step that should change the screen. Import credentials from 'e2e'.

A second walk over /todos, /controls, and /board covered press, long_press, double_tap, hover, a radio, a switch, a listbox, scroll_to, scroll (reason), drag, and back. The testbed's Sale banner has a hidden twin with the same text and test id, and printed await screen.getByText("Sale", { visible: true }).tap();.

The real proof: both drafts pasted into test files as is, plus one hand-written expect each, run green with the built CLI:

 ✓ target "web" command ready 88ms
 ✓ |web| tests/draft.e2e.ts (1 test) 858ms
 ✓ |web| tests/verbs.e2e.ts (1 test) 2.16s

 Test Files  2 passed (2)
      Tests  2 passed (2)

packages/e2e/tests/integration/mcp-test-lines.test.ts does the same in CI on a fixture page: it drives the built server, checks every line (a container scope, an .nth(1) between plain-div twins, a shadow-root twin the locator engine lists second, a test-id-only icon button, and the password by credential), then writes the draft into a test and runs it to passed.

iOS simulator (Settings app, a simulator I booted and shut down): lines print and replay.

>>> tap {"target":"n74"} 4974ms
Tapped #n74.
Test line: await screen.getByRole("button", "General").tap();

>>> tap {"target":"n756"} 3221ms
Tapped #n756.
Test line: await screen.getByRole("listitem", "Keyboard").tap();

 ✓ mobile engine for target "ios" prepared 706ms
 ✓ |ios| tests/settings.e2e.ts (1 test) 5.69s
   ✓ the drafted settings walk replays 5.69s

One locate there took 110 to 341 ms (5 samples), which is what each try adds per action; a control with a unique role and name costs one. Not run on an Android emulator.

A/B harness, author-journey (Sonnet, n=2 per arm; main rows are the shared baselines):

Run	Wall min	e2e runs	Sessions	MCP calls	Tool calls	Model steps	Cost $
main-author-1	0.8	1	1	19	26	17	0.16
main-author-2	0.9	1	1	17	23	14	0.14
branch-1 (first commit)	0.7	1	1	17	25	15	0.19
branch-2 (first commit)	0.6	1	1	17	21	15	0.18
branch-3 (head)	0.7	1	1	17	25	15	0.21
branch-4 (head)	0.8	1	1	17	25	15	0.19

Every branch arm wrote the locators the actions printed, word for word: getByRole('textbox', 'Username'), getByRole('combobox', 'Team'), and so on. Both main arms wrote getByLabel(...) for the fields instead. main-author-1 also spent one locate call and one extra navigate; no branch arm called locate.
The walk itself didn't change: all six arms walk the flow once over MCP, write the test after the walk, and pass on the first run. None called test_lines; on a three-screen flow the agent copied the lines from the results.
The cost gap is in the first two model steps: the branch arms wrote about 17k more tokens to the prompt cache, because the MCP server's instructions and tool descriptions changed, so the cached prefix main runs had warmed didn't match. By the end of the walk the branch arms also carried about 2k more cached input, the printed lines. Wall time is within noise at n=2.
The testbed is small enough that main's Sonnet authors this flow in one run, so this shows behavior (the lines get used) and no regression, not a speedup on a large app.

Testbed default suite: 79 passed, 1 skipped (on a port-0 copy of the config, since another agent held 4271).

Review

A fresh-context reviewer ran two rounds. Fixed:

A twin was placed by its box from the observation alone, so a page that shifted could make .nth() pin the wrong one. Now placement by position needs the twins where the observation saw them (each in place, or all moved by one offset).
Round two found the locator engine lists shadow-root matches after light-DOM ones while observe lists them in place, so "same count" didn't mean "same order". Twins in their old boxes in another order are now matched by box, and only a match placed for certain lends its test id. The fixture page covers it.
test_lines calls were folded into other in telemetry; the set now comes from the reserved names.
The tools and call descriptions didn't mention lines; ids are looked up on the newest screen only; the probe count per target is capped at 10.
A test ties the wrapped tools to GRAMMAR_TOOL_NAMES, so a new verb must get a line or a reason.

Dismissed: the claim that TargetSession.read accepts observation refs (the session adapter answers only located refs, packages/e2e/src/engine/session.ts), so the target's live box can't be read directly before the action.

#1317 feat(mcp): show the test line for each action in a session

Verified

Review

feat(mcp): show the test line for each action in a session