docs(skills): fix a failed test from its trace page instead of walking the flow again
After a red run, a coding agent tends to open an MCP session and click through the whole flow again to look at the screen that failed, then fix one line and rerun. The run already recorded that screen. The skill now tells the agent to fix from what the run printed and wrote, and to check the lines after the failing step before it reruns.
What changed
skills/e2e/SKILL.md: a new "Workflow: fixing a failed test" section after "Workflow: writing a test", whose step 5 now points to it.- Read the terminal's failure block (code frame,
expected/observed,❯ at,❯ on screen) and the trace page it names (❯ trace,## Screen at failure). - Fix every line that evidence covers, not only the failing one.
- A test stops at its first failed step, so check the lines after it before rerunning: against the screen at failure, the app source, or a live session.
- For a screen the evidence and the source leave out, go straight to it in a session: on the web,
navigateto the path of the❯ atURL or the screen's route. On a device, take the shortest path from launch. Never walk the flow from the start. - Rerun.
- Read the terminal's failure block (code frame,
skills/e2e/references/debugging.md, "Read the failure": item 1 names the terminal markers. A new item 2 holds the same rule and says what each field means for the fix. Items 3 to 5 are renumbered.skills/e2e/references/mcp.md, workflow step 4: fix from the evidence, and go straight to a screen it leaves out.docs/coding-agents.mdx, "How an agent reads a failure": two short paragraphs that say the same for readers (242 to 250 lines, 917 to 994 prose words).- A
.changesetentry,e2e: patch, because the skill ships in the package.
The field names come from what the CLI prints today: ❯ at, ❯ on screen, and ❯ trace (packages/e2e/src/report/list.ts:1035-1040), plus ## Screen at failure and Closest to what the locator asked for: (packages/e2e/src/report/failure-text.ts:555-569).
Decisions
- The rule lives in
SKILL.md, not only indebugging.md. In the main baselines, the agent fixing the seeded test openedwriting-tests.md(the routing table sends "Writing or fixing a test" there) and never openeddebugging.md.SKILL.mdis the one file every agent reads.debugging.mdrepeats the rule becausenpx e2e guide debugginghas to stand on its own. - The skill says to navigate to the path of the
❯ atURL. With a port-0 app URL, the run's port is gone by the time a session opens. - I didn't touch the cache, timeout, or session-closing guidance; other PRs own it. Step 5 says "topic
mcpsays when a session must close first" and doesn't restate the rule, so a change to that rule won't conflict with this PR. docs/debugging.mdxis unchanged. It has no section for coding agents.- I left the routing table alone to keep merges with the other skill PRs small.
Verified
Ran it locally: yes
node packages/e2e/dist/cli/bin.js guideandguide debuggingprint the new text.pnpm packofpackages/e2eships it inskills/e2e/anddocs/coding-agents.mdx.- The docs page renders in
mint dev:
A/B harness
I used the skill A/B harness: a fresh OpenCode agent (anthropic/claude-sonnet-5-5) with the e2e skill and MCP server, in a scratch testbed project.
fix-journeyseeds a test with three mistakes: a wrong label on/forms, a wrong button on step 1 of/wizard, and a wrong summary text. Each run surfaces only the next one.fix-journey-no-sourceis the same task plus "don't read files outside this project". The agent then can't grep the app's HTML, which a large app's source wouldn't allow either.main-fix-*are the shared baselines.b-main-fixnosrc-*ran the base commit's skill.b-*-1,2ran this PR's first wording,3,4the second, and5,6the final one.- "e2e cmds" is the harness count of commands that contain
e2e run. In four arms, one of those was a BSDsed -ithat failed before&&, so the run never started. "runs" counts the ones that executed.
| Run | Wall min | e2e cmds | runs | Sessions | MCP calls | Tool calls | Model steps | Cost $ | Final exit |
|---|---|---|---|---|---|---|---|---|---|
| main-fix-1 | 1.7 | 4 | 4 | 0 | 0 | 14 | 12 | 0.10 | 0 |
| main-fix-2 | 1.1 | 2 | 2 | 0 | 0 | 14 | 11 | 0.06 | 0 |
| b-fix-1 | 1.0 | 3 | 2 | 0 | 0 | 14 | 11 | 0.10 | 0 |
| b-fix-2 | 0.9 | 2 | 2 | 0 | 0 | 12 | 9 | 0.09 | 0 |
| b-fix-3 | 1.2 | 3 | 2 | 0 | 0 | 15 | 11 | 0.11 | 0 |
| b-fix-4 | 1.0 | 3 | 2 | 0 | 0 | 14 | 11 | 0.10 | 0 |
| b-fix-5 | 1.0 | 3 | 2 | 0 | 0 | 11 | 10 | 0.09 | 0 |
| b-fix-6 | 1.1 | 2 | 2 | 0 | 0 | 14 | 11 | 0.11 | 0 |
| b-main-fixnosrc-1 | 1.6 | 4 | 4 | 0 | 0 | 11 | 7 | 0.08 | 0 |
| b-main-fixnosrc-2 | 1.5 | 4 | 4 | 0 | 0 | 10 | 6 | 0.08 | 0 |
| b-fixnosrc-1 | 1.6 | 4 | 4 | 0 | 0 | 11 | 10 | 0.09 | 0 |
| b-fixnosrc-2 | 1.6 | 4 | 4 | 0 | 0 | 11 | 7 | 0.09 | 0 |
| b-fixnosrc-3 | 1.7 | 3 | 3 | 1 | 8 | 18 | 15 | 0.13 | 0 |
| b-fixnosrc-4 | 1.5 | 4 | 4 | 0 | 0 | 11 | 7 | 0.08 | 0 |
| b-fixnosrc-5 | 1.6 | 3 | 3 | 1 | 8 | 18 | 14 | 0.12 | 0 |
| b-fixnosrc-6 | 1.5 | 4 | 4 | 0 | 0 | 11 | 10 | 0.09 | 0 |
What the agents did, from events.jsonl:
- On
fix-journey, all six branch arms read## Screen at failureafter the first red run. Then they read the app source for the screens past it and fixed all three lines before the second run, which passed. They said why, in their own words: "Let me check the app source for the remaining screens, so I can fix everything in one pass" (b-fix-1), "check the wizard page, which the failure screen doesn't cover" (b-fix-4). On main, one of two arms did that (main-fix-2). The other fixed one line per run and needed 4 runs. - On
fix-journey-no-source, both main arms and both first-wording arms fixed one line per run, from each run's screen at failure, and opened no session. The first wording said to look live "only when none of that shows what a line needs", and the agents read that as only the failing line. With the later wording, 2 of 4 arms (b-fixnosrc-3 and -5) opened a session after the second run failed on/wizard. They callednavigate {url: "/wizard"}directly, with no sign-in and no/forms. Then they tapped through the wizard's two inner steps, which have no URL, and fixed the button and the summary text before the next run, which passed. - No arm walked the flow from the start. Every arm passed in the end.
- Wall time didn't move. On this app, a failing run costs 30 s on a wrong locator or 6 s on a wrong assertion, about what the session walk costs. Fewer runs, and fixes per run, are the behavior change. This doesn't show a speedup on a large app, where a rerun costs more and the agent can't read the source.