feat(test): add test.fixme

SA
1h ago
+552-80

Adds test.fixme, Playwright's marker for a test that is known broken, so a ported suite keeps those tests visible as skipped instead of deleting them or rewriting each one as test.skip.
Reports carry it as its own skip cause, fixme, so a consumer can count known-broken tests apart from tests skipped on purpose.

test.fixme('exports a PDF', async ({ app }) => {});            // registered skipped, body never runs
test('switches orgs', async ({ platform }) => {
  test.fixme(platform === 'android', 'crashes on Android');   // skips the running test
});

What and why

  • test.fixme takes exactly the shapes test.skip takes: (title, fn) at registration, and (), (reason), (condition, reason) inside a body or beforeEach. test.fixme(false) does nothing.
  • report-v1 gets a new skip cause, fixme, beside explicit (packages/e2e/schema/report-v1.schema.json, SkipInfo in packages/core/src/run/records.ts). The reason text is fixme, or fixme: <reason> for a reason that is not blank, so the list, junit, markdown, and HTML output read [fixme: crashes on Android] with no reporter change.
  • Both producers set the cause: a registered fixme test gets the registration mode fixme, which selection turns into the cause (packages/e2e/src/collect/select.ts), and the runtime forms throw a RuntimeSkip carrying the cause from RUNTIME_SKIP_APIS (packages/e2e/src/internal/skip.ts). A RuntimeSkip from an older copy of e2e, which has no cause, reports explicit.
  • Every consumer that treated explicit as a skip the test gave itself now treats fixme the same way, through a SelfSkipCause type: serial member accounting and Skipped After Failure in core, and the merge summary in @e2e-dev/github. --last-failed leaves fixme tests out, as it does explicit ones. NO_TESTS counts them as marked fixme.
  • A { skip: 'fixme' } option, on a test or a group, stays cause explicit. Only test.fixme sets the cause. The Playwright guide says so in the test.describe.fixme() row, and its test.fixme(title, fn) row says a serial group takes skip on the group instead, where Playwright accepts the registration.
  • The runtime forms reuse skipRunningTest, which now takes the API name, so a call outside a body or in a setup test names test.fixme(), not test.skip(). The test.skip messages are unchanged and now pinned in runner-lifecycle.test.ts.
  • The registration form goes through the shared registrar and registerOne (packages/e2e/src/collect/registry.ts), and so does test.skip(title, fn) now. In a serial group, each fails with test.fixme() cannot be used inside a serial group, ... or test.skip() cannot be used inside a serial group, ..., as skipIf and runIf do. Before, test.skip gave the generic test option "skip" cannot be overridden inside a serial group.
  • NO_TESTS used to say skipped with test.skip for any explicit skip, including skipIf and the skip option. It now says skipped explicitly (packages/e2e/src/collect/select.ts).
  • Docs: docs/reference/test.mdx, the Playwright guide (moved from "doesn't have yet" to "Map each API"), the docs/ja and docs/zh copies of both, and skills/e2e/references/writing-tests.md with its plugin copy. The docs have no table of skip causes, so the fixme cause is documented in the test.fixme section. Types are pinned in packages/e2e/tests/types/sdk-types.ts, including rejected forms such as test.fixme(title, options, fn).

Open questions

  • An older @e2e-dev/github installed beside this e2e counts a serial member that marks itself fixme as not executed, as it would for any new cause. I left the peer range alone.
  • A test's own reason wins over its group's: describe('g', { skip: 'tracked in #42' }, () => test.fixme('x', fn)) reports cause fixme and reason fixme. This is the existing layering in select.ts.
  • Out of scope, and still listed under "doesn't have yet": test.describe.fixme (use { skip: 'fixme' } on the group) and Playwright's (title, details, body) form. The parseModifierCall falsy-condition fix goes in a separate PR, as agreed.

Issue

Part of #1855 (Test runner row, test.fixme('not yet', fn)).

Verified

Ran it locally: yes

  • testbed, built CLI, a scratch file with a registered fixme, a runtime fixme with a reason, a plain test.skip, a { skip: 'fixme' } option, and a passing test:
   ↓ |web| tests-scratch/fixme-cause.e2e.ts > registered fixme [fixme]
   ↓ |web| tests-scratch/fixme-cause.e2e.ts > option fixme [fixme]
   ↓ |web| tests-scratch/fixme-cause.e2e.ts > runtime fixme with a reason [fixme: flaky on webkit]
   ↓ |web| tests-scratch/fixme-cause.e2e.ts > plain skip [not here]
   ✓ |web| tests-scratch/fixme-cause.e2e.ts > runs 8ms
      Tests  1 passed | 4 skipped (5)

report.json for that run:

["registered fixme"] {"cause":"fixme","reason":"fixme"}
["runtime fixme with a reason"] {"cause":"fixme","reason":"fixme: flaky on webkit"}
["plain skip"] {"cause":"explicit","reason":"not here"}
["option fixme"] {"cause":"explicit","reason":"fixme"}
["runs"] undefined
  • The error paths, same CLI:
tests-scratch/fixme-verify.e2e.ts: test.fixme() cannot be used inside a serial group, which skips as one unit; set skip on the serial describe instead
tests-scratch/skip-serial.e2e.ts: test.skip() cannot be used inside a serial group, which skips as one unit; set skip on the serial describe instead
tests-scratch/fixme-verify.e2e.ts: test.fixme(condition, reason) skips the running test and must be called inside a test body; to mark a test fixme at collection, register it with test.fixme(title, fn)
9 tests were collected but none is runnable: 8 file not selected by a positional argument, 1 marked fixme; pass --pass-with-no-tests to allow this
  • test.fixme in a test.each row body (#1882), on the branch merged with main 0e9d58ec:
   ✓ |web| tests-scratch/fixme-each.e2e.ts > row a 766ms
   ↓ |web| tests-scratch/fixme-each.e2e.ts > row b [fixme: only b]
["row b"] skipped {"cause":"fixme","reason":"fixme: only b"}
  • On the branch merged with main (0e9d58ec): pnpm check passes, including docs translations: 180 of 180 translated pages match their English outline and code. The e2e unit suite has 4161 passed, 2 skipped, and 2 failed: the two secret-ledger "in about its own size" timing tests, at a load average of 12, which pass 91/91 run alone. runner-lifecycle, skip-after-failure, and expected-failure pass (55/55), and so do @e2e-dev/github (42/42) and @e2e-dev/html-reporter (22/22).
  • The default testbed suite passes (593 passed, 3 skipped) on the branch merged with main 759d1fc2.
  • Reviewed with sol-loop: 7 rounds with gpt-6.1-sol. Sol verdict: approved

🤖 Generated with Claude Code