feat(test): add test.fixme
SA
1h ago+552+552+012345678901234567890123456789-80-80-01234567890123456789
Adds test.fixme, Playwright's marker for a test that is known broken, so a ported suite keeps those tests visible as skipped instead of deleting them or rewriting each one as test.skip.
Reports carry it as its own skip cause, fixme, so a consumer can count known-broken tests apart from tests skipped on purpose.
test.fixme('exports a PDF', async ({ app }) => {}); // registered skipped, body never runs
test('switches orgs', async ({ platform }) => {
test.fixme(platform === 'android', 'crashes on Android'); // skips the running test
});
What and why
test.fixmetakes exactly the shapestest.skiptakes:(title, fn)at registration, and(),(reason),(condition, reason)inside a body orbeforeEach.test.fixme(false)does nothing.- report-v1 gets a new skip cause,
fixme, besideexplicit(packages/e2e/schema/report-v1.schema.json,SkipInfoinpackages/core/src/run/records.ts). The reason text isfixme, orfixme: <reason>for a reason that is not blank, so the list, junit, markdown, and HTML output read[fixme: crashes on Android]with no reporter change. - Both producers set the cause: a registered fixme test gets the registration mode
fixme, which selection turns into the cause (packages/e2e/src/collect/select.ts), and the runtime forms throw aRuntimeSkipcarrying the cause fromRUNTIME_SKIP_APIS(packages/e2e/src/internal/skip.ts). ARuntimeSkipfrom an older copy of e2e, which has no cause, reportsexplicit. - Every consumer that treated
explicitas a skip the test gave itself now treatsfixmethe same way, through aSelfSkipCausetype: serial member accounting and Skipped After Failure in core, and the merge summary in@e2e-dev/github.--last-failedleaves fixme tests out, as it does explicit ones.NO_TESTScounts them asmarked fixme. - A
{ skip: 'fixme' }option, on a test or a group, stays causeexplicit. Onlytest.fixmesets the cause. The Playwright guide says so in thetest.describe.fixme()row, and itstest.fixme(title, fn)row says a serial group takesskipon the group instead, where Playwright accepts the registration. - The runtime forms reuse
skipRunningTest, which now takes the API name, so a call outside a body or in a setup test namestest.fixme(), nottest.skip(). Thetest.skipmessages are unchanged and now pinned inrunner-lifecycle.test.ts. - The registration form goes through the shared
registrarandregisterOne(packages/e2e/src/collect/registry.ts), and so doestest.skip(title, fn)now. In a serial group, each fails withtest.fixme() cannot be used inside a serial group, ...ortest.skip() cannot be used inside a serial group, ..., asskipIfandrunIfdo. Before,test.skipgave the generictest option "skip" cannot be overridden inside a serial group. NO_TESTSused to sayskipped with test.skipfor any explicit skip, includingskipIfand theskipoption. It now saysskipped explicitly(packages/e2e/src/collect/select.ts).- Docs:
docs/reference/test.mdx, the Playwright guide (moved from "doesn't have yet" to "Map each API"), thedocs/jaanddocs/zhcopies of both, andskills/e2e/references/writing-tests.mdwith its plugin copy. The docs have no table of skip causes, so thefixmecause is documented in thetest.fixmesection. Types are pinned inpackages/e2e/tests/types/sdk-types.ts, including rejected forms such astest.fixme(title, options, fn).
Open questions
- An older
@e2e-dev/githubinstalled beside thise2ecounts a serial member that marks itself fixme as not executed, as it would for any new cause. I left the peer range alone. - A test's own reason wins over its group's:
describe('g', { skip: 'tracked in #42' }, () => test.fixme('x', fn))reports causefixmeand reasonfixme. This is the existing layering inselect.ts. - Out of scope, and still listed under "doesn't have yet":
test.describe.fixme(use{ skip: 'fixme' }on the group) and Playwright's(title, details, body)form. TheparseModifierCallfalsy-condition fix goes in a separate PR, as agreed.
Issue
Part of #1855 (Test runner row, test.fixme('not yet', fn)).
Verified
Ran it locally: yes
- testbed, built CLI, a scratch file with a registered fixme, a runtime fixme with a reason, a plain
test.skip, a{ skip: 'fixme' }option, and a passing test:
↓ |web| tests-scratch/fixme-cause.e2e.ts > registered fixme [fixme]
↓ |web| tests-scratch/fixme-cause.e2e.ts > option fixme [fixme]
↓ |web| tests-scratch/fixme-cause.e2e.ts > runtime fixme with a reason [fixme: flaky on webkit]
↓ |web| tests-scratch/fixme-cause.e2e.ts > plain skip [not here]
✓ |web| tests-scratch/fixme-cause.e2e.ts > runs 8ms
Tests 1 passed | 4 skipped (5)
report.json for that run:
["registered fixme"] {"cause":"fixme","reason":"fixme"}
["runtime fixme with a reason"] {"cause":"fixme","reason":"fixme: flaky on webkit"}
["plain skip"] {"cause":"explicit","reason":"not here"}
["option fixme"] {"cause":"explicit","reason":"fixme"}
["runs"] undefined
- The error paths, same CLI:
tests-scratch/fixme-verify.e2e.ts: test.fixme() cannot be used inside a serial group, which skips as one unit; set skip on the serial describe instead
tests-scratch/skip-serial.e2e.ts: test.skip() cannot be used inside a serial group, which skips as one unit; set skip on the serial describe instead
tests-scratch/fixme-verify.e2e.ts: test.fixme(condition, reason) skips the running test and must be called inside a test body; to mark a test fixme at collection, register it with test.fixme(title, fn)
9 tests were collected but none is runnable: 8 file not selected by a positional argument, 1 marked fixme; pass --pass-with-no-tests to allow this
test.fixmein atest.eachrow body (#1882), on the branch merged with main0e9d58ec:
✓ |web| tests-scratch/fixme-each.e2e.ts > row a 766ms
↓ |web| tests-scratch/fixme-each.e2e.ts > row b [fixme: only b]
["row b"] skipped {"cause":"fixme","reason":"fixme: only b"}
- On the branch merged with main (
0e9d58ec):pnpm checkpasses, includingdocs translations: 180 of 180 translated pages match their English outline and code. Thee2eunit suite has 4161 passed, 2 skipped, and 2 failed: the twosecret-ledger"in about its own size" timing tests, at a load average of 12, which pass 91/91 run alone.runner-lifecycle,skip-after-failure, andexpected-failurepass (55/55), and so do@e2e-dev/github(42/42) and@e2e-dev/html-reporter(22/22). - The default testbed suite passes (593 passed, 3 skipped) on the branch merged with main
759d1fc2. - Reviewed with sol-loop: 7 rounds with gpt-6.1-sol. Sol verdict: approved
🤖 Generated with Claude Code