feat(e2e): test.fail for tests expected to fail

Merged
OK
13d ago
+955-48

Adds test.fail, so known gaps can be marked inline and the run fails once a gap closes.

test.fail('shows the cart badge', async ({ app }) => {});                 // register
test('gap on web', async ({ platform }) => {
  test.fail(platform === 'web', 'tap on a display: contents element');    // runtime, does not throw
});
test('exports a PDF', { fail: 'PDF export is broken' }, async () => {});  // option
  • Marked test that fails with a product failure (assertion, action, thrown error, timeout): passed. The attempt keeps expectedFailure: { reason, error }; the result gets expectedFailure: { reason }.
  • Marked test that passes: failed with new code UNEXPECTED_PASS, which names the reason. Not retried. An expected failure is not retried either.
  • Not covered: infrastructure, configuration, and internal errors (ENGINE_FAILURE included, also when one lands beside the test failure outside cleanup), interrupts, and blocked agent verdicts (STEP_BUDGET_EXHAUSTED and similar). A Firefox ENGINE_FAILURE tap stays a failure. Covering it would hide browser crashes; the right fix is for the engine to report that tap as NOT_ACTIONABLE.
  • A skip wins, also after a failure. Setup tests and serial members cannot be marked (INVALID_ARGUMENT); fail on a serial member or a describe is a COLLECTION_ERROR.
  • Report: additive optional fields in report-v1.schema.json. List, markdown, JUnit, trace page, MCP open_session text, and telemetry (tests_failed_as_expected) show expected failures.
  • Differs from Playwright: timeouts count as expected failures, and an unexpected pass is not retried.

Verified

Ran it locally: yes

  • testbed, scratch file using every form (not committed), e2e run tests/zz-scratch-fail.e2e.ts: exit 1, and retries: 2 on the unexpected pass ran only one attempt.
   ✓ |web| tests/zz-scratch-fail.e2e.ts > register form: heading is wrong (failed as expected) 715ms
     → expected to fail: ASSERTION_FAILED: expect.toBeVisible failed
   ✓ |web| tests/zz-scratch-fail.e2e.ts > option form: known gap (failed as expected) 633ms
     → todos page has no Archive button: LOCATOR_NOT_FOUND: locator matched no …
   ✓ |web| tests/zz-scratch-fail.e2e.ts > runtime form: web only gap (failed as expected) 597ms
     → title differs on web: ASSERTION_FAILED: expect.toHaveText failed
   ✓ |web| tests/zz-scratch-fail.e2e.ts > runtime condition false: normal test 63ms
   × |web| tests/zz-scratch-fail.e2e.ts > gap closed: passes unexpectedly 66ms
     → test is marked test.fail ("playground heading was missing") but passed; …
   ↓ |web| tests/zz-scratch-fail.e2e.ts > skip wins [not today]

report.json, trimmed:

{"t":"option form: known gap","status":"passed","expectedFailure":{"reason":"todos page has no Archive button"},"attempts":[{"status":"passed","ef":{"reason":"todos page has no Archive button","error":"LOCATOR_NOT_FOUND"}}]}
{"t":"gap closed: passes unexpectedly","status":"failed","expectedFailure":{"reason":"playground heading was missing"},"attempts":[{"status":"failed","error":"UNEXPECTED_PASS","ef":{"reason":"playground heading was missing"}}]}
  • Fresh-context review: fixed skip beside a failure (schema-invalid report), blocked agent verdicts, and infra errors in secondaryErrors being covered, plus trace-vs-video retention on an unexpected pass. Not changed: test.fail() called in afterEach applies under e2e run, but is refused in an MCP session's teardown.