#26068 cmux next: see what the agent does: a live view of the app in its chat, and every screenshot in Agent Activity

While an agent used an app through cmux computer use, nothing in its chat showed what it was doing. The person had to open Agent Activity, and that page showed only small thumbnails with no way to step through them. Seen on 2026-10-27 on macOS 26.6. The Codex app shows a live view of the app beside the chat. This PR adds a live view of the app to the chat, and a frame viewer for every screenshot to Agent Activity.

Live view in the chat

• Card (webviews/src/agent-session/acpmux/computerUse/ComputerUsePreviewCard.tsx): the app's window alone at the chat's top right (no card around it; a drop shadow that follows the window's rounded corners), as wide as the frame (at most 180 px). The app's name is in its label for screen readers. A small dark close button is always on the window's top left, with hover and press states. It eases in, the done state dims the window and pops in the app icon and then the check, and it eases out on close or auto-hide (cardMotion.ts: the last card stays drawn until its exit ends; with motion off it goes at once; a new frame replaces the old one only after it decodes). With the Inspector open, it sits 8 px below it (inspector/underInspector.ts). Click opens Agent Activity on the session. Panes under 600 px show no card.
• Done: the last frame dims under the app's icon with a green check. A done card hides itself 30 s later unless the agent acts again first (ComputerUseLivePreview.autoHideDelay). Unlike a close, a later act of the same session (the driver keeps it open between turns) shows it again.
• No cover: while the card shows, the chat column's right edge moves in (--cv-inset-end, mirrored in model.ts). A wide pane already has the room, so nothing moves there.
• Live video (ComputerUseLiveCapture.swift, ComputerUseFrameRing.swift, SharedMemoryWord.swift): while the session runs, the app asks the legacy helper for a window stream with its host-only application_surface_start (6 frames per second, long side 360 px). The helper writes the frames into a shared-memory ring; the app maps it read only and reads the newest finished slot, off the main thread. One request is in flight at a time; a tick that finds one running is skipped, never queued. It is not a tool call, so the helper records nothing in Agent Activity, and it never raises the window.
• Only while seen (ComputerUsePreviewSight.swift): the stream runs only while the person can see the chat. Hiding the chat, covering or minimizing its window, the session's end, the chat's turn ending, a close or the setting off stop the stream at once. A helper restart drops the old stream and starts one on the new helper.
• Fallback: a start the helper refuses, or a stream that fails, falls back to the frames the helper records per action, read over its socket (ComputerUseLivePreviewSource.swift). Their black window corners are cleared (ComputerUseFrameCorners.swift). An action's frame newer than the stream still loads, so a stalled stream (a minimized window) never freezes the card. The ring read checks its seqlock after an acquire fence.
• The agent's cursor (ComputerUseAgentCursorWindows.swift, cursorArt.tsx): the window stream has no cursor, and agents click by accessibility element, which reports no click point. The helper draws each agent cursor in a small window of its own, centered on the tip, so each frame tick also reads that window's place and the target window's bounds (no Screen Recording needed) and sends the tip as a fraction of the window. The card draws the helper's own cursor: its dart traced from the helper's cursor window, the cmux gradient (#12c7f5, #2d8cff at 53%, #6c5cff), a white rim, at the window's scale (never under 8 px). It glides 150 ms as the helper's does and plays a ripple on each click event. A pixel click's click_point is the fallback with no live cursor.
• App name and icon (ComputerUsePreviewApp.swift): looked up by pid or bundle id off the main thread, at most 32 cached.
• Which chat (ComputerUsePreviewRoute.swift, ComputerUsePreviewBoard.swift): the chat of the session's acp_session, then its harness session id, else the one working chat of the same agent family. When several chats could own it, no card shows. Several helper sessions of one turn feed one card, newest frame by capture time.
• Setting: computerUse.livePreview, on by default, in Settings > Computer Use ("Live Preview in Chat").

Every screenshot in Agent Activity

• Frame viewer (webviews/src/agent-activity/viewer.tsx): the selected session's screenshots, fitted in a well, with Previous and Next, Left, Right, Home and End, the step that made each frame, its time and "4 of 12", and a filmstrip of every frame. A frame in the timeline or a grid tile opens the viewer at that frame. The viewer loads only the shown frame and its neighbours; thumbnails load only in view.
• Stored bytes (AgentActivityPageService.swift, AgentActivityRequests.swift): the page gets the helper's JPEG as a data URL, with no decode and re-encode on the main actor. It asks for the full size only when the session keeps full frames, else the thumbnail, and falls back to the thumbnail. One open request selects a session and a frame in one turn.

Was stacked on #26060 (merged); the base is merged in.

Replaces #25912.

Changelog

• Added: while an agent uses an app on your Mac, its chat shows a small live view of that app with the agent's cursor in it, and the app's icon with a check when done.
• Added: Agent Activity steps through every screenshot of a session, with a filmstrip.

Tests

• ComputerUseLiveCaptureTests.swift: frames flow at the frame rate while the session runs and its chat is seen; a chat nobody sees starts no stream until it shows; a tick while a request runs is skipped; hiding the chat, a covered or minimized window, the session's end, the turn's end, a close and the setting off each stop the stream; a helper restart moves the stream to the new helper; no helper stops the pacing; a refused start and a failed read fall back to the per-action frames; an action's frame that loads after a live frame does not take the card back; the live target is the running, seen session that acted last; the wire reads an event's target window; the ring reads the newest finished slot scaled down and refuses another layout; a live frame is a PNG with the window's corners clear.
• ComputerUsePreviewBoardTests.swift: which chat shows a session (ids, guess, several working chats show no card, a named chat that is not open shows nothing), a tab that switches chat drops its card, two sessions of one turn feed one card in capture order, live and done states, close, setting off, frame notices read no chats.
• ComputerUsePreviewAppTests.swift: name fallback order, icon only when done, the wire's pid and window title, a real lookup of Calculator; a done card hides itself after 30 s, a later act of the same session shows it again (it stayed hidden before), a new action cancels the hide, a close leaves nothing scheduled, one chat's hide leaves another chat's card.
• Cursor: ComputerUsePreviewBoardTests (a running card carries the cursor only in its window and never when done; the live cursor wins and an accessibility click plays; the cursor window's center as a fraction; only click tools play), ComputerUseLiveCaptureTests.theAgentCursorFollowsTheHelpersCursorWindow (it shows, moves over a still window and goes), computerUsePreview.test.ts and ComputerUsePreviewCard.test.tsx (pointer read and checked, drawn at the window's scale, fixed size without a width, none without a cursor).
• ComputerUseFrameCornersTests.swift: black window corners cleared, a frame with no black corners and an app with a black edge left as they are.
• AgentActivityFramesTests.swift: the session record says what it stores; full frames are asked for only where the helper keeps them; the helper's bytes reach the page untouched; a missing full frame falls back to the thumbnail; a missing frame is an error; opening a frame selects its session and that frame in one request.
• webviews/src/agent-session/acpmux/computerUse/*.test.ts(x): running and done cards, the label with and without a name, the close button, the icon and check when done, close, the column's reserve math.
• webviews/src/agent-session/acpmux/inspector/underInspector.test.ts: the card's place under the open Inspector and the chat's shift.
• webviews/test/agent-activity-viewer.test.tsx: every event with a screenshot is a frame, in order; stepping by buttons and keys stops at the ends, and keys typed in the filter do not step; the filmstrip and the timeline open a frame; only the shown frame and its neighbours load; a session with no screenshots says so; a grid tile opens its newest screenshot.

Observed / Not run

• Ran locally (Xcode 26.6) on the head: swift test --package-path Packages/macOS/CmuxNext --filter 'ComputerUse|AgentActivity|PermissionRequestPresenter': 161 tests in 25 suites passed.
• bun test src/agent-session/acpmux/computerUse/ src/agent-session/acpmux/inspector/ test/agent-activity-viewer.test.tsx: 30 passed. bun run check, step by step: all pass except check:type-scale, vp fmt --check, vp lint and acpmux-protocol:check, which fail the same way on base 2c480abe5cf in files this PR does not change.
• python3 scripts/cmux-next/prepush-gate.py: no new violations.

Seen live by Lucas in the tagged app cuall (all four PRs of this stack together), on macOS 26.6:

• An agent ran Calculator through computer use; the card showed the app while it worked.
• Live video, stopwatch test: the card went from 00:03.56 to 00:05.42 in 1.6 s.
• 2026-10-28: the card's cursor followed the agent's clicks in Calculator, the same dart and colors as on screen.

Also run on 2026-10-28 (after review fixes and the cursor): swift test --filter 'ComputerUsePreviewBoardTests|ComputerUseLiveCaptureTests|ComputerUsePreviewAppTests': 77 passed; bun test src/agent-session/acpmux/computerUse: 22 passed; bun run check: passed.

Not run: Codex and other harnesses (only Claude was tried), several computer use sessions at once, the helper v2. Known limits: the card draws the default cursor gradient (a per-session color, if the helper ever uses one, is not read); with two cursors in one window it shows one; a guessed route for an agent of unknown family can still pick the wrong chat until the helper attributes sessions.

No capture: the live preview card needs a running Computer Use helper session with macOS privacy grants, which no capture scene or tour drives. Screenshots and a short recording of the card and the viewer from a real Mac follow.