fix(cli): report a skipped test in one shard only

Merged
JA
9d ago
+90-3

What and why

  • --shard cuts its slices from the runnable tests only, and applyShard (packages/e2e/src/collect/select.ts:776) left every skipped pair ({ skip }, test.skip(...) at declaration, a missing capability) untouched. So each shard kept every skipped test, and e2e list --shard, the list summary, junit.xml, summary.md, and report.json of every shard listed it again. A matrix of 3 shards over 3 tests and 1 skipped one reports 6 results instead of 4, and anything that adds the shards' JUnit files up counts the skip 3 times.
  • Now a skipped test is reported by the shard that runs the test before it in report order, the first shard when none comes before. The cut itself is unchanged: slices still differ by at most one runnable test, and a skipped test still never makes a shard non-empty.
  • In the other shards the pair becomes filtered and keeps its own skip, the way a platforms miss already does, so NO_TESTS still counts it as skipped with test.skip and the "split across N shards" count does not grow.
  • Playwright also reports each skipped test in one shard, but counts skipped tests as items of the cut; here the cut stays runnable tests only, so the documented slice sizes hold.
// packages/e2e/src/collect/select.ts, applyShard (this branch)
let previousKept = shard.index === 1;
return pairs.map((pair) => {
  if (pair.test.kind !== 'test') return pair;
  if (pair.disposition === 'skip') return previousKept ? pair : { ...pair, disposition: 'filtered' as const };
  if (pair.disposition !== 'run') return pair;
  previousKept = kept.has(shardItem(pair));
  ...

Verified

Ran it locally: yes

  • Scratch project, built CLI, one file with one, two, three, and later ({ skip: 'not ready yet' }), e2e run --shard i/3 --reporter list,junit:
shardmainthis branch
1/31 passed | 1 skipped (2), junit tests="2" skipped="1"1 passed (1), junit tests="1" skipped="0"
2/31 passed | 1 skipped (2), junit tests="2" skipped="1"1 passed (1), junit tests="1" skipped="0"
3/31 passed | 1 skipped (2), junit tests="2" skipped="1"1 passed | 1 skipped (2), junit tests="2" skipped="1"
  • The shards' report.json files validate against report-v1.schema.json, and their summary.selected now add up to the unsharded run's.
  • New unit test in packages/e2e/tests/unit/select.test.ts (a skipped test first, in the middle, and last, over 2 shards) fails on main and passes here.

Checklist

  • Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
  • Breaking change: title carries !, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md).
  • Docs updated in the same PR: the docs/**/*.mdx page for the behavior, and skills/e2e/ if the skill describes it.
  • Contract change: emitted .d.ts reviewed, tests/types/sdk-types.ts updated; wire change edits the schema and both fixtures.
  • Engine contract change (e2e/engine): changesets for e2e, @e2e-dev/web, and @e2e-dev/mobile.