fix(report): name the step whose error ended the test

Merged
TH
9d ago
+226-26

Linear: ENG-988

  • The markdown summary, the GitHub comment, and the trace page named the first step that did not pass. A step failure the test caught in try/catch became the lead, its step clause, and the Look at: line, while the code, message, and details were the later step's, the one that ended the test.
  • failedStepOf (packages/e2e/src/report/failure-text.ts) now starts from the attempt error. It keeps the steps that did not pass in the error's phase with its code. When the error names the line it unwound through, it takes the step whose own error names that line, else the last of them, since a caught failure comes earlier. A line none of them ran on means the error came from no step, as a value matcher's does, and the lead names none. When no step has the code, it takes the step a timeout or interrupt cut.
  • No wire change. The attempt error and the propagating step's error are the same serialized error, so the report already has what it needs, and a report.json written by a current runner renders right too.
  • Unit fixtures put the error on the failing step, as the runner does. A new integration case runs a caught waitFor and a failing assertion through the built runner and checks the page and summary.md; the soft-assertion suite now checks its summary names no step and points at the first soft matcher's line.
const alike = candidates.filter(({ step }) => step.error?.code === error.code);
const line = error.source;
if (line !== undefined && alike.length > 0) {
  return alike.findLast(({ step }) => step.error?.source?.file === line.file && step.error.source.line === line.line);
}
return alike.at(-1) ?? candidates.findLast(({ step }) => step.status === 'cancelled');

Verified

Ran it locally: yes

  • testbed, e2e run --reporter list,markdown on a scratch test (not committed) that catches a failing locator.waitFor at line 8 and ends on a failing toHaveCount at line 15, six steps in all:

    try {
      await screen.getByText('Nothing here').waitFor({ timeout: 300 });
    } catch {}
    await screen.getByRole('button', 'Add').tap();
    await expect(screen.getByTestId('todo')).toHaveCount(1);
    await expect(screen.getByTestId('todo')).toHaveCount(5, { timeout: 500 });
    
  • The new case in packages/e2e/tests/integration/traces.test.ts fails on main's dist with at step 2 of 4: locator.waitFor ... and passes here.

mainthis branch
summary.md lead**ASSERTION_FAILED** at step 3 of 6: locator.waitFor getByText("Nothing here") → visible, after 303ms**ASSERTION_FAILED** at step 6 of 6: expect.toHaveCount getByTestId("todo"), after 502ms
summary.md locationtests-scratch/eng988.e2e.ts:8tests-scratch/eng988.e2e.ts:15
trace page Look at:tests-scratch/eng988.e2e.ts:8tests-scratch/eng988.e2e.ts:15
trace page step liststeps 3 and 6 marked ✗same