docs(skills): write tests as one describe per feature, arrange-act-assert, one behavior each
Merged
KA
5d ago+464+464+012345678901234567890123456789-180-180-012345678901234567890123456789
The e2e skill told agents to check each agent.act outcome right after it. That produced long tests that chain several behaviors with checks in between, so a failing title said little about what broke. The skill now writes tests in one shape:
- One top-level
describeper unit or feature (test.setupstays top-level, since the collector rejects it inside adescribe). - Arrange, act, assert, separated by blank lines, every
expect/agent.assertat the end. - One behavior per test, named by its title. Several assertions are fine when they check that one outcome; a second behavior is a second test that repeats the setup in its arrange block.
- An arrange block built from
agent.actgoals ends with a wait for its state (locator.waitFor(),agent.waitFor). It is not an assertion, but it fails a broken precondition where it broke, and it confirms the setup goals for the replay cache, so a failing assertion no longer evicts them (settleStagedTracesinpackages/e2e/src/cache/context.tskeeps only entries before the last confirming step).
describe('todos', () => {
beforeEach(async ({ app }) => {
await app.open('/todos');
});
test('marks a todo as done', async ({ agent, screen }) => {
await agent.act('add a todo named {title}', { params: { title: 'Write the release notes' } });
await screen.getByRole('checkbox', 'Write the release notes').waitFor();
await agent.act('mark the todo as done');
await expect(screen.getByRole('status', 'Remaining')).toHaveText('0 remaining');
});
});
Where it lands:
skills/e2e/references/writing-tests.md: new "Structure" section; every example reshaped.skills/e2e/SKILL.md: quick start, the writing workflow, the rules (naming every check that confirms a recording: locator and engine matchers,agent.assert, and the waits), and the rerun advice.agent.md,mcp.md(the live walk splits a second behavior into a second test; setup tests stay top-level),bug-bash.md(the repro closes its setup with anagent.waitFor, so a broken sign-in is not confirmed as the defect),debugging.md.skills/migrate-to-e2e: a port still translates each source step literally, then arranges the lines in this shape before the first run. A source test that checks several behaviors splits into one test per behavior; a source assertion that only guarded a precondition becomes the wait closing the arrange block. Every source assertion still lands in some test, so the strictness count holds.docs/core-concepts.mdx(opening example and the goals section),docs/cache.mdx,docs/self-healing.mdx: the "check after each act" advice, so the docs agree with the skill they ship with.plugins/e2e/regenerated; patch changeset fore2e(migrate-to-e2eis not shipped in the package).
Verified
Ran it locally: yes
- A fresh agent with no context read only
SKILL.mdandwriting-tests.mdand wrote a deterministic todos suite for the testbed app. It produced onedescribe('todos'), four single-behavior tests, and arrange/act/assert blocks with everyexpectlast. Run with the built CLI against the testbed app (a throwaway config on port 4299, since another testbed held 4271):
✓ target "web" command at http://127.0.0.1:4299/ ready 89ms
✓ |web| tests/tmp-verify/todos.e2e.ts > todos > adds a todo to the list 1.75s
✓ |web| tests/tmp-verify/todos.e2e.ts > todos > marks a todo as done 170ms
✓ |web| tests/tmp-verify/todos.e2e.ts > todos > deletes a todo from the list 231ms
✓ |web| tests/tmp-verify/todos.e2e.ts > todos > shows only completed todos under the Done filter 264ms
Test Files 1 passed (1)
Tests 4 passed (4)
- A fresh agent with only the migrate skill's step 5, the Structure section, and the Playwright guide ported a one-test Playwright
checkout flow(5 assertions, checks between steps). It produced onedescribe('checkout')with three single-behavior tests (adds a book to the cart,updates the total when quantity changes,confirms the order at checkout), turned theShopheading check into abeforeEachwait, and mapped every source assertion to a final assertion. Not run against an app. - The cache claims were checked against
packages/e2e/src/cache/context.tsandrun/steps.tsin a fresh-context review;session,tags, andagentContexton adescribeagainstcollect/registry.ts. - Not run: the agentic examples (no model call was made for this change).