feat(test): add test.describe.fail and the fail option on describe

VI
6h ago
+335-79

What and why

  • Closes the test.describe.fail('legacy', fn) row of #1855: mark every test in a group expected to fail.
  • describe now takes the fail option, the way it already takes skip. describe.fail(title, fn) is shorthand for fail: true, as test.fail(title, fn) is for a test. A reason goes in the option.
  • A test reads fail from its groups as well as itself (packages/e2e/src/collect/select.ts:165). Any truthy layer marks it and the innermost reason wins. fail: false does not unmark, same as skip.
  • Each test is still judged on its own: a failure passes it, a pass fails it with UNEXPECTED_PASS.
  • Refused on, inside, or around a serial group, since a serial member cannot be marked.
describe.fail('legacy checkout', () => {
  test('pays by card', async ({ app, browser }) => {});   // expected to fail
});
describe('legacy refunds', { fail: 'refund API not ported yet' }, () => {});

Issue

Part of #1855 (Test runner: test.describe.fail('legacy', fn)).

Verified

Ran it locally: yes

A small page in Chromium through the built CLI (node packages/e2e/dist/cli/bin.js run): the legacy card payment is broken, the coupon was fixed, refunds were never ported.

 ✓ |web| tests/checkout.e2e.ts > legacy checkout > pays by card (failed as expected) 2.45s
     → expected to fail: ASSERTION_FAILED: expect.toHaveText failed
 × |web| tests/checkout.e2e.ts > legacy checkout > applies a coupon 131ms
     → test is marked test.fail ("expected to fail") but passed; remove the mar…
 ✓ |web| tests/checkout.e2e.ts > legacy refunds > refunds an order (failed as expected) 2.16s
     → refund API not ported yet: ASSERTION_FAILED: expect.toHaveText failed
 ✓ |web| tests/checkout.e2e.ts > modern checkout pays 123ms
  • Exit code 1 from the unexpected pass. report.json carries each result's expectedFailure reason, and none on the test outside the groups.
  • A serial group inside describe.fail stops the run before anything executes (exit 2): a serial group cannot sit inside a group marked fail; its members cannot be expected to fail.

Checklist

  • Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
  • Breaking change: title carries !, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md).
  • Docs updated in the same PR: the docs/**/*.mdx page for the behavior, and skills/e2e/ if the skill describes it.
  • Contract change: emitted .d.ts reviewed, tests/types/sdk-types.ts updated; wire change edits the schema and both fixtures.
  • Engine contract change (e2e/engine): changesets for e2e, @e2e-dev/web, and @e2e-dev/mobile.

🤖 Generated with Claude Code