#14928 Dogfood the app from CI with JSON tours

Summary

Agents can now drive the built app on a CI runner and look at what happened, with no per-tour compile. You write a JSON tour and run:

scripts/run-e2e.sh --scenario dogfood/scenarios/sidebar-and-chrome-tour.json --ref <sha> --frames

You get the tour's screenshots, contact sheets, accessibility trees, and socket replies under $TMPDIR/cmux-ui-frames/<run>/. This builds on the frames tool from #14916.

How it works

• cmuxUITests/DogfoodScenarioUITests reads the tour at run time and executes the steps:
  • socket requests, which can save results and reuse their fields in later steps
  • key presses and typing
  • clicks and hovers, targeted by accessibility id, label, or window point
  • menu paths, waits, and existence checks
• shot and tree steps attach a screenshot or the accessibility tree with .keepAlways, so they survive a passing test.
• A failed step is recorded along with a screenshot, and the tour continues. The test fails at the end with the list.
• Without a scenario the test skips, so ordinary UI runs are unaffected.
• test-e2e.yml gains a dogfood_scenario input (base64). The e2e action forwards it to the runner as TEST_RUNNER_CMUX_DOGFOOD_SCENARIO_B64, which is required because sudo -u resets the environment.
• Scenario runs add their dispatch id to the concurrency group, so several tours of one commit run side by side instead of cancelling each other.
• dispatch-focused-test.py --scenario:
  • validates and encodes the tour and defaults the selector to the dogfood test
  • skips the already-failed/running guard, since every tour shares one selector
  • refuses a tour past the dispatch-input size limit
• e2e-frames.py now also exports text attachments (trees, socket replies, steps.log) next to the frames.
• Guide: skills/cmux-testing/references/dogfood-scenarios.md, linked from the cmux-testing skill. The starter tour is in dogfood/scenarios/.

Testing

• The three new launcher tests in tests/test_run_e2e.py pass locally: scenario dispatch and encoding, a tour with no steps refused, and a selector or scenario required.
• A live tour of this branch follows (with --workflow-ref, since the input isn't on main yet). Its frames will be posted here.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Agents can now drive the built app on a CI runner with a JSON tour instead of compiling per tour: scripts/run-e2e.sh --scenario <tour>.json --ref <sha> --frames returns screenshots, contact sheets, accessibility trees, and socket replies.

DogfoodScenarioUITests

• Reads the tour at run time and executes socket requests, key presses, typing, clicks, hovers, menu paths, waits, and existence checks.
• The socket is enabled through launch environment overrides since the launch argument loses to the runner's settings file, and binds inside the runner's temporary directory because the sandboxed runner cannot connect(2) to /tmp (EPERM); paths the app reports in diagnostics are also tried.
• shot and tree steps keep their attachments, and a failed step is recorded with a screenshot while the tour continues, failing at the end with the full list.
• Shots crop to the app's front window by default, the window zooms at launch, and a final whole-display shot shows what covered the app on the shared CI desktop.
• Shell text goes in initial_input; initial_command replaces the shell, so quick commands show Ghostty's failed-launch screen.
• A launch landing in the background on headless runners is absorbed as an expected failure and activated explicitly; a 0-frame expected failure means the runner couldn't foreground the app, so dispatch the tour again.
• A silent socket attaches socket-diagnostics.txt with the app's diagnostics, socket files, and launch environment.
• Starter tours cover the sidebar and chrome, browser notifications, and the right sidebar with menus, context menus, and the terminal find bar.
• Skips when no scenario is set, so ordinary UI runs are unaffected.

Tooling

• test-e2e.yml takes a base64 dogfood_scenario input, forwarded to the runner as TEST_RUNNER_CMUX_DOGFOOD_SCENARIO_B64 since sudo -u resets the environment.
• The runner's dialog cleanup now uses SIGKILL and closes Diagnostics Reporter (matched by executable path, since its name outgrows pkill's 16-byte -x compare), Problem Reporter, and CoreServicesUIAgent (a launch-failure dialog covered a tour).
• Scenario runs add their dispatch id to the concurrency group so simultaneous tours of one commit don't cancel each other.
• dispatch-focused-test.py --scenario validates and encodes the tour, defaults the selector to the dogfood test, and skips the already-failed/running guard.
• e2e-frames.py exports text attachments (trees, socket replies, steps.log) next to the frames and skips XCUITest's per-event snapshots; contact sheets are one xstack per 12 frames since ffmpeg 9's tile filter wrote only the first frame.

Summary by CodeRabbit

• New Features
  • Added configurable UI tours for testing browser notifications, workspaces, the sidebar, and app controls.
  • UI tours can run in CI with screenshots and accessibility details, and report failed steps alongside supporting attachments.
  • CI test runs can now accept a tour scenario, including runs without a separate test filter.
• Documentation
  • Added guidance for creating and running UI tours in CI.