feat: write a failure page per failed test with each step's actions, polls, cache decision, and app log

Closed
OK
22d ago
+2091-429

Stacked on #822. When a test fails, a coding agent (or a person) should read one file and know what happened. Today that file is opt-in (--reporter markdown), deterministic steps record nothing (events: []), the cache's decision never leaves report.json, and console errors and failed requests exist only inside the trace zip the parent PR removes.

Every run now writes a page per failed or flaky test under .e2e/failures/, and each step on it says what it did.

flowchart LR
  E[web engine: console, pageerror, requestfailed, 4xx/5xx] -- appLog --> R[StepRecorder: redact, file under the step]
  L[LocatorEngine: node acted on] --> R
  P[expect / waitFor: sample history] --> R
  C[step cache: stop point, write outcome, entry] --> R
  R --> J[report.json events]
  J --> F[runner writes failures/*.md]
  F --> T["terminal: ❯ details failures/x.md"]
  • Engine hook: EngineAttemptContext.appLog({ source, level, text }). The session adapter checks each entry, holds it until a step recorder takes it, and files it under the step that started last, redacted then clipped (200 per attempt). The web engine reports console errors and warnings, uncaught exceptions, failed requests, and 4xx/5xx responses (packages/web/src/app-log.ts).
  • Deterministic steps: an engine event per action names the node from the resolve's own cache, no engine call (tap button "Add" (appeared after 1200ms, went stale 1x)); a point action names its point (tap at (120, 48)). expect and waitFor record a poll event with the readings in order: one value the whole time is a wrong expectation, changing values are a race.
  • Cache: step.cache.entry (the file), detail (at action 2 of 3, tap button "Apply": ..., or the recorded end state that never showed), and the report-only write (saved, kept, unconfirmed, evicted, no-change). New entries store their key's context, so a rekeyed REPLAY_STALE names what changed: since the app's identity (app.identity, else its URL) or environment changed. Existing entries are not rewritten to gain it.
  • Also recorded: the hook a step ran in (step.phase), loop-guard and wind-down notes on the agent's turns, and the message of a failed agent action.
  • Output: the runner writes the pages before run-finished (named <file>-<title slug>-<16 hex of the result id>.md) and names them in the event and FinishedRun.failurePages. The markdown reporter keeps only summary.md, linked to them. A page that cannot be written is a stderr line, like junit.xml.

Verified

Ran it locally: yes

  • testbed, a scratch config with a scripted model (no provider key) and four failing tests: agentic, agent step then a wrong expectation, a missing locator, and a page that logs errors. e2e run:
 FAIL  |web| tests-audit/audit.e2e.ts > app: the page breaks while the test runs
LOCATOR_NOT_FOUND: locator did not become visible: getByRole("heading", name: "Todos")
 ❯ at http://127.0.0.1:4388/nope
 ❯ app 2 errors, 2 warnings logged
 ❯ details .e2e-audit/failures/audit-app-the-page-breaks-while-the-796fac5f.md
...
   Failures  .e2e-audit/failures/ (4 pages: steps, cache, app log, screen)

The page's steps:

1. ✓ `app.open` `/todos` (37ms in beforeEach) — `tests-audit/audit.e2e.ts:4`
2. ✓ `browser.evaluate` (3ms) — `tests-audit/audit.e2e.ts:24`
   - ✗ console error: `failed to load todos: TypeError: items is undefined`
3. ✓ `app.open` `/nope` (5ms) — `tests-audit/audit.e2e.ts:30`
   - ✗ uncaught error: `RangeError: render loop exceeded at eval (...)`
   - ⚠ network warning: `GET /api/missing 404 Not Found`
   - ⚠ network warning: `GET /nope 404 Not Found`
4. ✗ `locator.waitFor` `getByRole("heading", name: "Todos") → visible` (1.0s) — **LOCATOR_NOT_FOUND**
   - waitFor gave up after 11 reads in 1.0s: absent x11

and for the agent step that passed before a wrong expectation:

2. ✓ `agent.act` "Add a todo named Milk." (420ms, 2 model calls)
   - cache: no replay: nothing recorded for this step yet; recording saved (`.e2e/cache/1227d5a6….json`)
   - type "Milk" into textbox "New todo" (18ms)
   - press "Enter" on the focused field (4ms)
5. ✗ `expect.toHaveText` `getByRole("status", name: "Remaining")` (1.6s) — **ASSERTION_FAILED**
   - expect gave up after 16 reads in 1.6s: text "0 remaining" (1 match) x16
  • Second run of the same suite: cache: replayed all 2 recorded actions, no model call; recording kept. Changing the app port and running with --strict-cache: REPLAY_STALE ... since the app's identity (app.identity, else its URL) or environment changed after it was recorded.
  • web-benchmark agentic suite, CI=1 --strict-cache with committed recordings and no key: 26 replayed. The 7 failures are steps that call the model by design, and the failure pages said so (no replay: a retry never replays, at action 2 of 3, tool type (run-time value)).