docs(skills): write tests as one describe per feature, arrange-act-assert, one behavior each

Merged
KA
5d ago
+464-180

The e2e skill told agents to check each agent.act outcome right after it. That produced long tests that chain several behaviors with checks in between, so a failing title said little about what broke. The skill now writes tests in one shape:

  • One top-level describe per unit or feature (test.setup stays top-level, since the collector rejects it inside a describe).
  • Arrange, act, assert, separated by blank lines, every expect / agent.assert at the end.
  • One behavior per test, named by its title. Several assertions are fine when they check that one outcome; a second behavior is a second test that repeats the setup in its arrange block.
  • An arrange block built from agent.act goals ends with a wait for its state (locator.waitFor(), agent.waitFor). It is not an assertion, but it fails a broken precondition where it broke, and it confirms the setup goals for the replay cache, so a failing assertion no longer evicts them (settleStagedTraces in packages/e2e/src/cache/context.ts keeps only entries before the last confirming step).
describe('todos', () => {
  beforeEach(async ({ app }) => {
    await app.open('/todos');
  });

  test('marks a todo as done', async ({ agent, screen }) => {
    await agent.act('add a todo named {title}', { params: { title: 'Write the release notes' } });
    await screen.getByRole('checkbox', 'Write the release notes').waitFor();

    await agent.act('mark the todo as done');

    await expect(screen.getByRole('status', 'Remaining')).toHaveText('0 remaining');
  });
});

Where it lands:

  • skills/e2e/references/writing-tests.md: new "Structure" section; every example reshaped.
  • skills/e2e/SKILL.md: quick start, the writing workflow, the rules (naming every check that confirms a recording: locator and engine matchers, agent.assert, and the waits), and the rerun advice.
  • agent.md, mcp.md (the live walk splits a second behavior into a second test; setup tests stay top-level), bug-bash.md (the repro closes its setup with an agent.waitFor, so a broken sign-in is not confirmed as the defect), debugging.md.
  • skills/migrate-to-e2e: a port still translates each source step literally, then arranges the lines in this shape before the first run. A source test that checks several behaviors splits into one test per behavior; a source assertion that only guarded a precondition becomes the wait closing the arrange block. Every source assertion still lands in some test, so the strictness count holds.
  • docs/core-concepts.mdx (opening example and the goals section), docs/cache.mdx, docs/self-healing.mdx: the "check after each act" advice, so the docs agree with the skill they ship with.
  • plugins/e2e/ regenerated; patch changeset for e2e (migrate-to-e2e is not shipped in the package).

Verified

Ran it locally: yes

  • A fresh agent with no context read only SKILL.md and writing-tests.md and wrote a deterministic todos suite for the testbed app. It produced one describe('todos'), four single-behavior tests, and arrange/act/assert blocks with every expect last. Run with the built CLI against the testbed app (a throwaway config on port 4299, since another testbed held 4271):
 ✓ target "web" command at http://127.0.0.1:4299/ ready 89ms
   ✓ |web| tests/tmp-verify/todos.e2e.ts > todos > adds a todo to the list 1.75s
   ✓ |web| tests/tmp-verify/todos.e2e.ts > todos > marks a todo as done 170ms
   ✓ |web| tests/tmp-verify/todos.e2e.ts > todos > deletes a todo from the list 231ms
   ✓ |web| tests/tmp-verify/todos.e2e.ts > todos > shows only completed todos under the Done filter 264ms
 Test Files  1 passed (1)
      Tests  4 passed (4)
  • A fresh agent with only the migrate skill's step 5, the Structure section, and the Playwright guide ported a one-test Playwright checkout flow (5 assertions, checks between steps). It produced one describe('checkout') with three single-behavior tests (adds a book to the cart, updates the total when quantity changes, confirms the order at checkout), turned the Shop heading check into a beforeEach wait, and mapped every source assertion to a final assertion. Not run against an app.
  • The cache claims were checked against packages/e2e/src/cache/context.ts and run/steps.ts in a fresh-context review; session, tags, and agentContext on a describe against collect/registry.ts.
  • Not run: the agentic examples (no model call was made for this change).