docs(skills): fix a failed test from its trace page instead of walking the flow again

Merged
KA
6d ago
+145-28

After a red run, a coding agent tends to open an MCP session and click through the whole flow again to look at the screen that failed, then fix one line and rerun. The run already recorded that screen. The skill now tells the agent to fix from what the run printed and wrote, and to check the lines after the failing step before it reruns.

What changed

  • skills/e2e/SKILL.md: a new "Workflow: fixing a failed test" section after "Workflow: writing a test", whose step 5 now points to it.
    1. Read the terminal's failure block (code frame, expected/observed, ❯ at, ❯ on screen) and the trace page it names (❯ trace, ## Screen at failure).
    2. Fix every line that evidence covers, not only the failing one.
    3. A test stops at its first failed step, so check the lines after it before rerunning: against the screen at failure, the app source, or a live session.
    4. For a screen the evidence and the source leave out, go straight to it in a session: on the web, navigate to the path of the ❯ at URL or the screen's route. On a device, take the shortest path from launch. Never walk the flow from the start.
    5. Rerun.
  • skills/e2e/references/debugging.md, "Read the failure": item 1 names the terminal markers. A new item 2 holds the same rule and says what each field means for the fix. Items 3 to 5 are renumbered.
  • skills/e2e/references/mcp.md, workflow step 4: fix from the evidence, and go straight to a screen it leaves out.
  • docs/coding-agents.mdx, "How an agent reads a failure": two short paragraphs that say the same for readers (242 to 250 lines, 917 to 994 prose words).
  • A .changeset entry, e2e: patch, because the skill ships in the package.

The field names come from what the CLI prints today: ❯ at, ❯ on screen, and ❯ trace (packages/e2e/src/report/list.ts:1035-1040), plus ## Screen at failure and Closest to what the locator asked for: (packages/e2e/src/report/failure-text.ts:555-569).

Decisions

  • The rule lives in SKILL.md, not only in debugging.md. In the main baselines, the agent fixing the seeded test opened writing-tests.md (the routing table sends "Writing or fixing a test" there) and never opened debugging.md. SKILL.md is the one file every agent reads. debugging.md repeats the rule because npx e2e guide debugging has to stand on its own.
  • The skill says to navigate to the path of the ❯ at URL. With a port-0 app URL, the run's port is gone by the time a session opens.
  • I didn't touch the cache, timeout, or session-closing guidance; other PRs own it. Step 5 says "topic mcp says when a session must close first" and doesn't restate the rule, so a change to that rule won't conflict with this PR.
  • docs/debugging.mdx is unchanged. It has no section for coding agents.
  • I left the routing table alone to keep merges with the other skill PRs small.

Verified

Ran it locally: yes

  • node packages/e2e/dist/cli/bin.js guide and guide debugging print the new text. pnpm pack of packages/e2e ships it in skills/e2e/ and docs/coding-agents.mdx.
  • The docs page renders in mint dev:

coding-agents

A/B harness

I used the skill A/B harness: a fresh OpenCode agent (anthropic/claude-sonnet-5-5) with the e2e skill and MCP server, in a scratch testbed project.

  • fix-journey seeds a test with three mistakes: a wrong label on /forms, a wrong button on step 1 of /wizard, and a wrong summary text. Each run surfaces only the next one.
  • fix-journey-no-source is the same task plus "don't read files outside this project". The agent then can't grep the app's HTML, which a large app's source wouldn't allow either.
  • main-fix-* are the shared baselines. b-main-fixnosrc-* ran the base commit's skill. b-*-1,2 ran this PR's first wording, 3,4 the second, and 5,6 the final one.
  • "e2e cmds" is the harness count of commands that contain e2e run. In four arms, one of those was a BSD sed -i that failed before &&, so the run never started. "runs" counts the ones that executed.
RunWall mine2e cmdsrunsSessionsMCP callsTool callsModel stepsCost $Final exit
main-fix-11.7440014120.100
main-fix-21.1220014110.060
b-fix-11.0320014110.100
b-fix-20.922001290.090
b-fix-31.2320015110.110
b-fix-41.0320014110.100
b-fix-51.0320011100.090
b-fix-61.1220014110.110
b-main-fixnosrc-11.644001170.080
b-main-fixnosrc-21.544001060.080
b-fixnosrc-11.6440011100.090
b-fixnosrc-21.644001170.090
b-fixnosrc-31.7331818150.130
b-fixnosrc-41.544001170.080
b-fixnosrc-51.6331818140.120
b-fixnosrc-61.5440011100.090

What the agents did, from events.jsonl:

  • On fix-journey, all six branch arms read ## Screen at failure after the first red run. Then they read the app source for the screens past it and fixed all three lines before the second run, which passed. They said why, in their own words: "Let me check the app source for the remaining screens, so I can fix everything in one pass" (b-fix-1), "check the wizard page, which the failure screen doesn't cover" (b-fix-4). On main, one of two arms did that (main-fix-2). The other fixed one line per run and needed 4 runs.
  • On fix-journey-no-source, both main arms and both first-wording arms fixed one line per run, from each run's screen at failure, and opened no session. The first wording said to look live "only when none of that shows what a line needs", and the agents read that as only the failing line. With the later wording, 2 of 4 arms (b-fixnosrc-3 and -5) opened a session after the second run failed on /wizard. They called navigate {url: "/wizard"} directly, with no sign-in and no /forms. Then they tapped through the wizard's two inner steps, which have no URL, and fixed the button and the summary text before the next run, which passed.
  • No arm walked the flow from the start. Every arm passed in the end.
  • Wall time didn't move. On this app, a failing run costs 30 s on a wrong locator or 6 s on a wrong assertion, about what the session walk costs. Fewer runs, and fixes per run, are the behavior change. This doesn't show a speedup on a large app, where a rerun costs more and the agent can't read the source.