UI tests: one labelled frame per action, built in CI; scripts/ui-test

Merged
TE
20d ago
+396-259

Seeing what a UI test did took a 100 MB xcresult download and a pile of unlabelled screenshots. Now CI does the work, and the result is organized by test action.

scripts/ui-test SidebarHelpMenuUITests     # run it in CI at your pushed HEAD, then show its steps
scripts/ui-test <run id or URL>            # show a finished run's steps

For each test it gives a steps.md like this (from run 36294794315):

Failure: XCTAssertTrue failed - Expected Settings to open from the sidebar Help menu

1. Start  (frames/01-start.jpg)
2. Open com.cmuxterm.app.debug  (frames/02-open-com-cmuxterm-app-debug.jpg)
3. Open com.cmuxterm.app.debug  (frames/03-open-com-cmuxterm-app-debug.jpg)
4. Click Any (First Match)  (frames/04-click-any-first-match.jpg)
5. Click "SidebarHelpMenuOptionSettings" MenuItem  (frames/05-click-...-menuitem.jpg)  <- failed here

It also writes one 960 px JPEG per action (the screen right after it) and 3x4 contact sheets.

What changed

  • Per action. e2e-frames.py reads the xcresult's activity tree. It keeps the last screenshot under each top-level action and names the frame after the action. It drops XCUITest's bookkeeping ("Checking existence of", "Wait for idle"), and it points "failed here" at the last action before XCTest's own failure snapshot. Named captures become capture: <name> steps, so passing tests show their captures. Screen recordings are sampled once a second, captioned with the action running at that moment.
  • In CI. test-e2e's UI runs build these after the tests. The action lists go to the job summary, failures first and expanded, and the frames go to a ui-frames artifact of a few MB (six tests came to 3.7 MB). The tool downloads that artifact and falls back to the xcresult for older runs. The script is added to the .e2e-workflow sparse checkout, so it comes from the workflow revision like the action does, and a bisect of an older commit still gets it. test_ci_e2e_compilation_cache.py now expects the two-entry sparse list.
  • One entry point. scripts/ui-test either runs a test by name (through dispatch-focused-test.py --frames) or reads a finished run by id or URL. The testing-skill guide is shorter and leads with it.
  • Fixes. sips exits 0 on a missing input and writes nothing; that frame is now skipped with a note. It happens: xcresulttool export sometimes lists a screenshot it never writes. Contact sheets are captioned only when ffmpeg has drawtext, which Homebrew's current ffmpeg lacks; tile N is still step N.

Verification

  • End to end on a mini (run 36316225276, workflow from this branch): CI built 20, 9 and 9 steps for the three Help tests, sampling their screen recordings with AVFoundation because the mini has no ffmpeg. It uploaded ui-frames (4.7 MB) and wrote the job summary. scripts/ui-test SidebarHelpMenuUITests then downloaded the artifact and printed the per-action lists. No contact sheets there without ffmpeg; frames and steps.md don't need it.
  • Locally, run 36294794315's xcresult (screenshots, 6 tests, 3.7 MB) and run 36312258398 through the xcresult fallback, with and without ffmpeg.
  • Subagent review: one blocker, now fixed. The upload's hashFiles() gate ignored runner.temp, so the upload never ran. Also fixed: a second --test query returned the first query's tests, and summary escaping, frame numbering and one-line titles.
  • Guards: tests/test_ci_e2e_compilation_cache.py passes the sparse-list assertion. One unrelated test fails identically on main under macOS bash 3.2.
  • PR CI: green.

🤖 Generated with Claude Code