cmux next: a live view of the app an agent uses, in its chat
Closed
LU
11d ago+3105+3105+0123456789012345678901234567890123456789-30-30-01234567890123456789
While an agent uses an app through cmux computer use, nothing in its chat showed what it was doing. The person had to open Agent Activity to see it. The Codex app shows a live view of the app beside the chat, and the app's icon with a check when the work is done. This PR adds that card to cmux NEXT chats.
- Card (
webviews/src/agent-session/acpmux/computerUse/ComputerUsePreviewCard.tsx): a glass card at the chat's top right with the app's name and its newest frame, as wide as the frame (at most 180 px). It uses the Chat Inspector's glass (one rule set ininspector/inspector.css). With the Inspector open, it sits 8 px below it, right-aligned, in both placements (inspector/underInspector.ts). It updates as the helper records frames. Done, the frame dims under the app's icon with a green check. Click opens Agent Activity. A close button hides it. Panes under 600 px show no card. - No cover: while the card shows, the chat column's right edge moves in (
--cv-inset-end, mirrored inmodel.tstranscriptColumn). Transcript rows, the composer and the bands under it follow. A wide pane already has the room, so nothing moves there. - Frames (
ComputerUseLivePreview,ComputerUseLivePreviewSource): the app follows the helper's activity sessions over its socket and reads the newest frame's thumbnail (activity_frame) off the main thread. The helper's frames are JPEGs, so a window's transparent rounded corners come out black.ComputerUseFrameCornersfinds the black corners and clears them to the window's outline (PNG). A screen capture, or an app whose own edge is black, stays as it is. - App name and icon (
ComputerUsePreviewApp): the legacy helper names only the pid. The app looks it up off the main thread (running app by pid, else Launch Services by bundle id) and caches at most 32 apps. Title order: the helper's name, the looked-up name, the window title, no title. The icon goes with the done state only. - Which chat (
ComputerUsePreviewRoute,ComputerUsePreviewBoard): the chat of the session'sacp_session, then of its harness session id. A session that names a chat that is not open shows nowhere. Until the helper stamps those ids, the route guesses the one working chat of the same agent family. When several chats could own it, no card shows. A route holds only while its tab still shows the same chat:chat.selecton the tab drops it at once. - One card per chat: one agent turn can make several helper sessions at once (one opens the app, another clicks). All of them feed the chat's card, which shows the newest frame by capture time. It turns done when all of them have ended or the chat's turn is over.
- Main thread: a frame notice reads no chats. Chats are read only when the session list, a carded chat's turn, or a tab's chat changes, and a loaded frame goes only to its chat.
- Setting:
computerUse.livePreview, on by default, in Settings > Computer Use ("Live Preview in Chat").
Stacked on #25831.
Changelog
- Added: while an agent uses an app on your Mac, its chat shows a small live view of that app, and the app's icon with a check when done.
Tests
Packages/macOS/CmuxNext/Tests/CmuxNextAppTests/ComputerUsePreviewBoardTests.swift: which chat shows a session (ids, guess, several working chats show no card, a named chat that is not open shows nothing), a tab that switches chat drops its card, two sessions of one turn feed one card in capture order, live and done states, close, setting off, frame notices read no chats.Packages/macOS/CmuxNext/Tests/CmuxNextAppTests/ComputerUsePreviewAppTests.swift: name fallback order, icon only when done, the wire's pid and window title, the full flow with a fake resolver, a real lookup of Calculator.Packages/macOS/CmuxNext/Tests/CmuxNextAppTests/ComputerUseFrameCornersTests.swift: black window corners cleared (corner alpha 0, edge and center 255), a frame with no black corners and an app with a black edge left as they are.webviews/src/agent-session/acpmux/computerUse/*.test.ts(x): layout reserve math, title fallback, close, card render.webviews/src/agent-session/acpmux/inspector/underInspector.test.ts: the card's place under the open Inspector and the chat's shift.
Observed / Not run
- Ran locally (Xcode 26.6):
swift test --package-path Packages/macOS/CmuxNext --filter 'ComputerUseFrameCornersTests|ComputerUsePreviewAppTests|ComputerUsePreviewBoardTests': 40 passed.bun test src/agent-session/acpmux/computerUse/: 11 passed.bun run check: all steps pass except the base format error insrc/pages/settings/store.ts. - Seen live in a tagged build with the computer use stack and #25901: a Claude chat computed 64 x 27 in Calculator through cmux computer use. The card showed "Calculator" and the frames while it worked, then the Calculator icon with the green check. The chat text stopped before the card.
- Ran the corner clearing on a real Calculator frame from the helper: radius 20.5 px found, corners cleared.
- Not run: Codex and other harnesses (only Claude was tried), several computer use sessions at once, and the Upstream helper.