docs(skills): start each test near the screen it covers

Merged
KA
6d ago
+152-1

Every run of a test, and every rerun while a coding agent fixes one line, walks through everything before the screen under test. This tells the agent to keep that part short.

  • skills/e2e/references/writing-tests.md: a new section, "Start near the screen under test", added to ## Contents:
    • sign in once in a test.setup and declare session. Sessions last one run, so the setup still runs every run, once per target for the selected tests that need it
    • app.open('/path') straight to the screen. observe over MCP prints the path
    • seed data through the app's API in a test.extend fixture that sends browser.cookies() (when the app signs in with a cookie) and parses the response with toMatchSchema, so a failed seed fails in the fixture, not 30 s later on a locator. Over MCP, a project tool does the same
    • on a device there are no sessions. device.openLink opens a deep link
    • grow a long test one screen at a time when its steps were not checked live. Verified agent.act steps replay from the cache, which keeps them across a later failure. That holds for a first attempt with a read-write cache, so grow outside CI and with --retries 0. Keep the title, goals, and params (unique() for run-unique data) unchanged so the keys still match
  • skills/e2e/SKILL.md workflow step 3: one sentence that points there.
  • docs/core-concepts.mdx, "Organize tests": the first bullet says the same briefly and replaces "Prepare its data in fixtures or hooks."
  • Changeset: e2e patch, since the skill ships in the package.

The claims rest on:

  • one setup unit per target and setup test: packages/e2e/src/run/units.ts:60. Setups run only for sessions a selected test needs: packages/e2e/src/collect/select.ts:499
  • the session is restored at launch, before fixtures exist: packages/e2e/src/run/execute.ts:630 and :1014
  • the cache keeps entries a check confirmed before a failure: settleStagedTraces, packages/e2e/src/cache/context.ts:188

Verified

Ran it locally: yes

  • node packages/e2e/dist/cli/bin.js guide writing-tests prints the new section.

  • The skill's fixture example, run in a scratch project against the testbed app with a small /api/projects endpoint added that requires the session cookie and JSON. The setup signed in, the fixture seeded, the test opened the deep link, and the teardown deleted the project:

     ✓ |web| tests/auth.setup.e2e.ts (1 test) 581ms
     ✓ |web| tests/example.e2e.ts (1 test) 203ms
    .e2e/logs/app.log: seeded project 411, deleted project 411
    

    Without session, the same test fails at the fixture line in 134 ms:

       × renames a project 134ms
    ASSERTION_FAILED: expected {"error":"signed out"} to match the schema:
    - id: Invalid input: expected string, received undefined
    

    The example also typechecks with and without the DOM lib.

  • One setup for three dependent tests, and selecting one dependent by file:line still runs its setup:

     ✓ |web| tests/auth.setup.e2e.ts (1 test) 304ms
     ✓ |web| tests/near.e2e.ts (4 tests) 320ms
    
    $ e2e run tests/near.e2e.ts:19
     ✓ |web| tests/auth.setup.e2e.ts (1 test) 328ms
     ✓ |web| tests/near.e2e.ts (1 test) 275ms
    
  • Not verified live: the cache claim behind "grow one screen at a time". No model key on this machine, so it rests on settleStagedTraces and tests/integration/agent-trace-cache.test.ts.

  • The docs bullet, rendered with mint dev (pnpm docs:check and scripts/check-docs-anchors.ts pass):

    Organize tests bullet

Skill A/B

A fresh OpenCode agent (anthropic/claude-sonnet-5-5) with the e2e MCP server authors tests in a scratch copy of the testbed app, once with main's skill and once with this branch's. n=2 per arm, so this is a sanity check, not a benchmark.

author-journey asks for one test covering sign-in, the profile form, and the workspace wizard. author-signed-in is a task I added: the same three flows, framed as things a signed-in admin does, with no "one test" constraint.

RunSkillTaskWall mine2e runsMCP sessionsMCP callsTool callsModel stepsCost $Final exitWhat it wrote
main-author-1mainjourney0.8111926170.160one test: sign-in form, then app.open('/forms'), app.open('/wizard')
main-author-2mainjourney0.9111723140.140same
e-author-journey-1branchjourney0.7111724160.200same
e-author-journey-2branchjourney0.7111723160.190same
e-main-author-signed-in-1mainsigned-in1.0111726140.200test.setup + session: 'admin', three tests opened by path
e-main-author-signed-in-2mainsigned-in0.7111723160.180same
e-author-signed-in-1branchsigned-in0.8111724150.210same
e-author-signed-in-2branchsigned-in0.7111726140.200same
  • No behavior change on this app, and no regression. Every agent on either skill already opened /forms and /wizard by path and passed on its first run.
  • On author-journey both branch agents read the new section (read references/writing-tests.md in both event streams) and still kept the sign-in form inside the one test, since the task asks for one test that includes the sign-in.
  • On author-signed-in, main's skill already leads to a setup test and a session, from the existing "Sign-in sessions" section. The branch writes the same files.
  • The branch journey arms cost $0.19-0.20 against $0.14-0.16 for the older main baselines. The main arms run next to the branch on the second task cost $0.18-0.20, so I read that as run-to-run variance, not the skill.
  • The testbed has no API to seed and the harness config has no model for agent.act, so the seeding and growing advice is not exercised. It pays off on a large app where the screen under test sits deep in a flow, which this harness cannot show.

Decisions

  • The docs bullet goes in core-concepts.mdx "Organize tests", not coding-agents.mdx: the advice holds for anyone who structures tests, and that section's example already starts with app.open('/billing'). The example in coding-agents.mdx ("How an agent writes and runs a test") still signs in inside the test. I left it alone.
  • "Keep the title and goals unchanged while the test grows" is close to what the fast-rerun-loop PR in this batch says. It stays here because growing a test pays off only while the earlier keys hold.
  • A fresh-context review ran twice. Fixed from it: the cache key sentence now names params and unique(); the fixture example sends a JSON content type and the session's cookies and checks the response; the web-only advice in SKILL.md, core-concepts.mdx, and the changeset now names the device path; the cookie line is limited to apps that sign in with a cookie; the "pays only for its new part" claim names the retry and CI exceptions.
  • Shortening pass on core-concepts.mdx: the edited bullet replaces one sentence, and nothing else on the page says it. 230 to 234 lines, 719 to 756 prose words. No other cuts in "Organize tests".