docs(skills): start each test near the screen it covers
Every run of a test, and every rerun while a coding agent fixes one line, walks through everything before the screen under test. This tells the agent to keep that part short.
skills/e2e/references/writing-tests.md: a new section, "Start near the screen under test", added to## Contents:- sign in once in a
test.setupand declaresession. Sessions last one run, so the setup still runs every run, once per target for the selected tests that need it app.open('/path')straight to the screen.observeover MCP prints the path- seed data through the app's API in a
test.extendfixture that sendsbrowser.cookies()(when the app signs in with a cookie) and parses the response withtoMatchSchema, so a failed seed fails in the fixture, not 30 s later on a locator. Over MCP, a project tool does the same - on a device there are no sessions.
device.openLinkopens a deep link - grow a long test one screen at a time when its steps were not checked live. Verified
agent.actsteps replay from the cache, which keeps them across a later failure. That holds for a first attempt with a read-write cache, so grow outsideCIand with--retries 0. Keep the title, goals, and params (unique()for run-unique data) unchanged so the keys still match
- sign in once in a
skills/e2e/SKILL.mdworkflow step 3: one sentence that points there.docs/core-concepts.mdx, "Organize tests": the first bullet says the same briefly and replaces "Prepare its data in fixtures or hooks."- Changeset:
e2epatch, since the skill ships in the package.
The claims rest on:
- one setup unit per target and setup test:
packages/e2e/src/run/units.ts:60. Setups run only for sessions a selected test needs:packages/e2e/src/collect/select.ts:499 - the session is restored at launch, before fixtures exist:
packages/e2e/src/run/execute.ts:630and:1014 - the cache keeps entries a check confirmed before a failure:
settleStagedTraces,packages/e2e/src/cache/context.ts:188
Verified
Ran it locally: yes
-
node packages/e2e/dist/cli/bin.js guide writing-testsprints the new section. -
The skill's fixture example, run in a scratch project against the testbed app with a small
/api/projectsendpoint added that requires the session cookie and JSON. The setup signed in, the fixture seeded, the test opened the deep link, and the teardown deleted the project:✓ |web| tests/auth.setup.e2e.ts (1 test) 581ms ✓ |web| tests/example.e2e.ts (1 test) 203ms .e2e/logs/app.log: seeded project 411, deleted project 411Without
session, the same test fails at the fixture line in 134 ms:× renames a project 134ms ASSERTION_FAILED: expected {"error":"signed out"} to match the schema: - id: Invalid input: expected string, received undefinedThe example also typechecks with and without the DOM lib.
-
One setup for three dependent tests, and selecting one dependent by
file:linestill runs its setup:✓ |web| tests/auth.setup.e2e.ts (1 test) 304ms ✓ |web| tests/near.e2e.ts (4 tests) 320ms $ e2e run tests/near.e2e.ts:19 ✓ |web| tests/auth.setup.e2e.ts (1 test) 328ms ✓ |web| tests/near.e2e.ts (1 test) 275ms -
Not verified live: the cache claim behind "grow one screen at a time". No model key on this machine, so it rests on
settleStagedTracesandtests/integration/agent-trace-cache.test.ts. -
The docs bullet, rendered with
mint dev(pnpm docs:checkandscripts/check-docs-anchors.tspass):
Skill A/B
A fresh OpenCode agent (anthropic/claude-sonnet-5-5) with the e2e MCP server authors tests in a scratch copy of the testbed app, once with main's skill and once with this branch's. n=2 per arm, so this is a sanity check, not a benchmark.
author-journey asks for one test covering sign-in, the profile form, and the workspace wizard. author-signed-in is a task I added: the same three flows, framed as things a signed-in admin does, with no "one test" constraint.
| Run | Skill | Task | Wall min | e2e runs | MCP sessions | MCP calls | Tool calls | Model steps | Cost $ | Final exit | What it wrote |
|---|---|---|---|---|---|---|---|---|---|---|---|
| main-author-1 | main | journey | 0.8 | 1 | 1 | 19 | 26 | 17 | 0.16 | 0 | one test: sign-in form, then app.open('/forms'), app.open('/wizard') |
| main-author-2 | main | journey | 0.9 | 1 | 1 | 17 | 23 | 14 | 0.14 | 0 | same |
| e-author-journey-1 | branch | journey | 0.7 | 1 | 1 | 17 | 24 | 16 | 0.20 | 0 | same |
| e-author-journey-2 | branch | journey | 0.7 | 1 | 1 | 17 | 23 | 16 | 0.19 | 0 | same |
| e-main-author-signed-in-1 | main | signed-in | 1.0 | 1 | 1 | 17 | 26 | 14 | 0.20 | 0 | test.setup + session: 'admin', three tests opened by path |
| e-main-author-signed-in-2 | main | signed-in | 0.7 | 1 | 1 | 17 | 23 | 16 | 0.18 | 0 | same |
| e-author-signed-in-1 | branch | signed-in | 0.8 | 1 | 1 | 17 | 24 | 15 | 0.21 | 0 | same |
| e-author-signed-in-2 | branch | signed-in | 0.7 | 1 | 1 | 17 | 26 | 14 | 0.20 | 0 | same |
- No behavior change on this app, and no regression. Every agent on either skill already opened
/formsand/wizardby path and passed on its first run. - On
author-journeyboth branch agents read the new section (read references/writing-tests.mdin both event streams) and still kept the sign-in form inside the one test, since the task asks for one test that includes the sign-in. - On
author-signed-in, main's skill already leads to a setup test and a session, from the existing "Sign-in sessions" section. The branch writes the same files. - The branch journey arms cost $0.19-0.20 against $0.14-0.16 for the older main baselines. The main arms run next to the branch on the second task cost $0.18-0.20, so I read that as run-to-run variance, not the skill.
- The testbed has no API to seed and the harness config has no model for
agent.act, so the seeding and growing advice is not exercised. It pays off on a large app where the screen under test sits deep in a flow, which this harness cannot show.
Decisions
- The docs bullet goes in
core-concepts.mdx"Organize tests", notcoding-agents.mdx: the advice holds for anyone who structures tests, and that section's example already starts withapp.open('/billing'). The example incoding-agents.mdx("How an agent writes and runs a test") still signs in inside the test. I left it alone. - "Keep the title and goals unchanged while the test grows" is close to what the fast-rerun-loop PR in this batch says. It stays here because growing a test pays off only while the earlier keys hold.
- A fresh-context review ran twice. Fixed from it: the cache key sentence now names params and
unique(); the fixture example sends a JSON content type and the session's cookies and checks the response; the web-only advice inSKILL.md,core-concepts.mdx, and the changeset now names the device path; the cookie line is limited to apps that sign in with a cookie; the "pays only for its new part" claim names the retry andCIexceptions. - Shortening pass on
core-concepts.mdx: the edited bullet replaces one sentence, and nothing else on the page says it. 230 to 234 lines, 719 to 756 prose words. No other cuts in "Organize tests".