feat(test): add test.describe.fail and the fail option on describe
VI
6h ago+335+335+012345678901234567890123456789-79-79-01234567890123456789
What and why
- Closes the
test.describe.fail('legacy', fn)row of #1855: mark every test in a group expected to fail. describenow takes thefailoption, the way it already takesskip.describe.fail(title, fn)is shorthand forfail: true, astest.fail(title, fn)is for a test. A reason goes in the option.- A test reads
failfrom its groups as well as itself (packages/e2e/src/collect/select.ts:165). Any truthy layer marks it and the innermost reason wins.fail: falsedoes not unmark, same asskip. - Each test is still judged on its own: a failure passes it, a pass fails it with
UNEXPECTED_PASS. - Refused on, inside, or around a serial group, since a serial member cannot be marked.
describe.fail('legacy checkout', () => {
test('pays by card', async ({ app, browser }) => {}); // expected to fail
});
describe('legacy refunds', { fail: 'refund API not ported yet' }, () => {});
Issue
Part of #1855 (Test runner: test.describe.fail('legacy', fn)).
Verified
Ran it locally: yes
A small page in Chromium through the built CLI (node packages/e2e/dist/cli/bin.js run): the legacy card payment is broken, the coupon was fixed, refunds were never ported.
✓ |web| tests/checkout.e2e.ts > legacy checkout > pays by card (failed as expected) 2.45s
→ expected to fail: ASSERTION_FAILED: expect.toHaveText failed
× |web| tests/checkout.e2e.ts > legacy checkout > applies a coupon 131ms
→ test is marked test.fail ("expected to fail") but passed; remove the mar…
✓ |web| tests/checkout.e2e.ts > legacy refunds > refunds an order (failed as expected) 2.16s
→ refund API not ported yet: ASSERTION_FAILED: expect.toHaveText failed
✓ |web| tests/checkout.e2e.ts > modern checkout pays 123ms
- Exit code 1 from the unexpected pass.
report.jsoncarries each result'sexpectedFailurereason, and none on the test outside the groups. - A serial group inside
describe.failstops the run before anything executes (exit 2):a serial group cannot sit inside a group marked fail; its members cannot be expected to fail.
Checklist
- Changeset added (
pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only. - Breaking change: title carries
!, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md). - Docs updated in the same PR: the
docs/**/*.mdxpage for the behavior, andskills/e2e/if the skill describes it. - Contract change: emitted
.d.tsreviewed,tests/types/sdk-types.tsupdated; wire change edits the schema and both fixtures. - Engine contract change (
e2e/engine): changesets fore2e,@e2e-dev/web, and@e2e-dev/mobile.
🤖 Generated with Claude Code