cmux NEXT: click-through check for picks that stick and still chat switches

Merged
TE
17d ago
+683-10

scripts/cmux-next/clickthrough-live.py clicks through a tagged cmux NEXT build the way a person does, and fails when a model, reasoning level or access mode doesn't stick, or when a chat switch is slow, blank or shifts its layout. It re-runs the two click-through passes behind #21677, #21685 and #21688, so a later PR that brings those bugs back gets caught.

./scripts/reload.sh --tag <tag> --build-only
python3 scripts/cmux-next/clickthrough-live.py --tag <tag>

Pass 1, picks stick. Picks Haiku 5.5 · Low on a New Tab, then a folder, then sends a typed prompt with a click on Send and answers Trust. It then checks:

  • what acpmux says the chat runs: the picked model and reasoning, in full access;
  • the chips afterwards;
  • a New Tab opened right after the pick: it names the pick from its first frame;
  • a second chat on Sonnet moved to High;
  • each chat after a switch back and forth;
  • both chats, and a new tab, after Quit and Keep Sessions and a relaunch, including that the first switch after the relaunch shows no blank pane.

Pass 2, switching. Makes --chats (default 12) chats in 3 workspaces, then switches through them twice with Cmd-Shift-] and Ctrl-Cmd-]. After each switch, the new debug.agent_pane layout_watch records the frames of the composer, its chips, the header and the transcript rows on every animation frame for 1.2 s. The check fails if:

  • first content takes longer than --first-content-ms (default 150);
  • anything moves, appears or scrolls after first paint;
  • the pane reads blank for more than a frame;
  • the sidebar's workspace order changes.

The chats run on the fake ACP agent registered as acpmux's claude harness, so the curated Claude models apply and no sign-in or tokens are needed. FAKE_PICKER_MODES=claude makes the fake report Claude Code's permission modes and a model option, and keep each session's model and effort. The app runs with its own ACPMUX_HOME, XDG_CONFIG_HOME and fixture folders and never activates. It refuses a bundle id outside com.cmuxterm.app.debug.*, and refuses to start while another copy of the tag runs (they would share its socket and daemons). The tag's daemons end with the script (tag_teardown.py). Results go to <out>/results.md, results.json and switches.json.

Changes outside the script, all debug or test only:

  • debug.agent_pane layout_watch (new layoutWatch.ts), and chat_state.composer: the chips as drawn.
  • fake_agent.py: the claude mode. The existing FAKE_PICKER=1 behavior is unchanged.

Verification

On cmux16, tagged build clk of feat-cmux-next c25d784 plus #21677, #21685 and #21688, merged locally; acpmux was swapped in from that tree because this host has no zig to build cmux-tui:

  • After review fixes (ea1116a9a9d: the chat a send started is found by its folder and prompt, a relaunch or pass 2 that measured too little fails instead of passing, blank time is a duration), a run with all the fixes passes every scenario except two. New Tab pages draw no access chip (being followed up separately). And after a relaunch, the first switch to the other chat read blank for 27–28 ms in one of two switches, which is a finding, not a check bug.
  • Three clean runs gave the same results. Pass 2 passed in each, at a median of 23–26 ms to first content (worst 34 ms) over 16 chat switches, with nothing moving, popping in or jumping. Pass 1 passed except for one scenario: New Tab pages draw no access chip. That's #21272's area, and it was reported to the composer-pickers lane.
  • Before acpmux carried #21677, two runs failed "a New Tab after a pick names the pick from its first frame" (Medium instead of Low), so the check catches that regression.
  • #21685's relaunch case: the manual repro failed before the fix; the check passes with it.
  • cargo test -p acpmux --test start_defaults --test start_mode (fake agent): pass. bun run typecheck: no errors.
  • verify-local.py --affected origin/feat-cmux-next --swift-changed origin/feat-cmux-next: 10 of 11 pass; localization parity fails with 34264 errors, the same count as on feat-cmux-next itself. Native compilation is covered by the tagged builds above.
  • Not run in CI or on the fleet; it needs a GUI host.

Changelog

none

🤖 Generated with Claude Code