feat(e2e): test.fail for tests expected to fail
Merged
OK
13d ago+955+955+012345678901234567890123456789-48-48-01234567890123456789
Adds test.fail, so known gaps can be marked inline and the run fails once a gap closes.
test.fail('shows the cart badge', async ({ app }) => {}); // register
test('gap on web', async ({ platform }) => {
test.fail(platform === 'web', 'tap on a display: contents element'); // runtime, does not throw
});
test('exports a PDF', { fail: 'PDF export is broken' }, async () => {}); // option
- Marked test that fails with a product failure (assertion, action, thrown error, timeout):
passed. The attempt keepsexpectedFailure: { reason, error }; the result getsexpectedFailure: { reason }. - Marked test that passes:
failedwith new codeUNEXPECTED_PASS, which names the reason. Not retried. An expected failure is not retried either. - Not covered: infrastructure, configuration, and internal errors (
ENGINE_FAILUREincluded, also when one lands beside the test failure outside cleanup), interrupts, and blocked agent verdicts (STEP_BUDGET_EXHAUSTEDand similar). A FirefoxENGINE_FAILUREtap stays a failure. Covering it would hide browser crashes; the right fix is for the engine to report that tap asNOT_ACTIONABLE. - A skip wins, also after a failure. Setup tests and serial members cannot be marked (
INVALID_ARGUMENT);failon a serial member or adescribeis aCOLLECTION_ERROR. - Report: additive optional fields in
report-v1.schema.json. List, markdown, JUnit, trace page, MCPopen_sessiontext, and telemetry (tests_failed_as_expected) show expected failures. - Differs from Playwright: timeouts count as expected failures, and an unexpected pass is not retried.
Verified
Ran it locally: yes
- testbed, scratch file using every form (not committed),
e2e run tests/zz-scratch-fail.e2e.ts: exit 1, andretries: 2on the unexpected pass ran only one attempt.
✓ |web| tests/zz-scratch-fail.e2e.ts > register form: heading is wrong (failed as expected) 715ms
→ expected to fail: ASSERTION_FAILED: expect.toBeVisible failed
✓ |web| tests/zz-scratch-fail.e2e.ts > option form: known gap (failed as expected) 633ms
→ todos page has no Archive button: LOCATOR_NOT_FOUND: locator matched no …
✓ |web| tests/zz-scratch-fail.e2e.ts > runtime form: web only gap (failed as expected) 597ms
→ title differs on web: ASSERTION_FAILED: expect.toHaveText failed
✓ |web| tests/zz-scratch-fail.e2e.ts > runtime condition false: normal test 63ms
× |web| tests/zz-scratch-fail.e2e.ts > gap closed: passes unexpectedly 66ms
→ test is marked test.fail ("playground heading was missing") but passed; …
↓ |web| tests/zz-scratch-fail.e2e.ts > skip wins [not today]
report.json, trimmed:
{"t":"option form: known gap","status":"passed","expectedFailure":{"reason":"todos page has no Archive button"},"attempts":[{"status":"passed","ef":{"reason":"todos page has no Archive button","error":"LOCATOR_NOT_FOUND"}}]}
{"t":"gap closed: passes unexpectedly","status":"failed","expectedFailure":{"reason":"playground heading was missing"},"attempts":[{"status":"failed","error":"UNEXPECTED_PASS","ef":{"reason":"playground heading was missing"}}]}
- Fresh-context review: fixed skip beside a failure (schema-invalid report), blocked agent verdicts, and infra errors in
secondaryErrorsbeing covered, plus trace-vs-video retention on an unexpected pass. Not changed:test.fail()called inafterEachapplies undere2e run, but is refused in an MCP session's teardown.