Play a video or recording an agent writes inline in the chat

Merged
TE
1mo ago
+893-10

Summary

A video or audio file an agent writes or links now plays inline in the chat. Before this, it showed only as a path chip.

  • Host check: the page asks media.load {src} with the path from the reply or tool output. The host checks it like a reply image: inside the session's folders, not on the deny list, a regular file, a known type, at most 4 GB. Video types are mp4, m4v, mov and webm; audio types are mp3, m4a, aac, wav and flac.
  • Granted URL: a checked file gets an unguessable cmux-agent://pane/__media/<token>.<ext> URL. The page never names a file itself.
  • Serving: the pane's scheme handler serves a granted file in byte ranges, up to 8 MB per response. It refuses a file that was swapped for a link after the grant. The page CSP adds media-src 'self'; connect-src stays 'none'.
  • Player: it mounts only once it nears the viewport and asks for metadata only, so a long transcript loads no media it doesn't show. A video plays inline with controls, muted while the pointer rests on it. Expand shows it large over the pane. Nothing opens a browser.
  • Where: tool output paths and reply images (![](demo.mp4)) use it. If the host refuses a file, it shows as a name card.

Web media (CI artifact URLs, GitHub user-attachments) is refused for now; it follows in its own PR. With the agent.pageHost tunable on, the page runs under the page host's CSP, which has no media-src, so video there falls back to the name card.

Targets feat-cmux-next (#18401 and #18357 landed as 7672e438688). Part of bead cx-ncc.51.

Testing

  • Red commit ceefe3b4a90, then the fix.
  • bun test src/agent-session/acpmux/chips/ReplyMedia.test.tsx (local): failed at the red commit because the module was missing; 3 pass after the fix.
  • bun test src/agent-session/acpmux src/gallery: 1197 pass. The 4 failures (composer pickers, chat menu, palette Continue in, automation chat menu) also fail on clean feat-cmux-next.
  • AgentPaneReplyMediaTests cover the grant (inside, outside, wrong type, web URL, same file same URL, unknown token), byte ranges (206 slice, open-ended, whole file, 416 past the end) and the params contract. Fleet Swift suite (step 18909ba90b44, run-swift-testing-suites.sh at c24deb96ef7): all three pass. The later merges changed no Swift.
  • After the base merge (head 41ccb71ebc1, base 6314b9888f7), local in webviews/: bun test src/agent-session/acpmux/chips src/agent-session/acpmux/conversation test/gallery-coverage.test.ts test/pane-english.test.ts test/ui-rules.test.ts 222 pass, 0 fail; the typecheck passes on all 1047 files. Fleet Swift suite on 41ccb71ebc1 (merged head): it compiles, and AgentPaneReplyImageTests and AgentPaneReplyMediaTests pass. A later base merge (9506c26b6cb, with #18423) brought only unrelated Swift (terminal sizing, the context-menu export fix); react-apps-check is green on 9506c26b6cb
  • vp check passes on the changed files. verify-local.py --only swift-syntax --swift-changed passes. The agent pane bundle is no longer committed (base dropped it); builds generate it.

Changelog

Added: A video or audio file an agent writes or links plays inline in the agent chat, with an Expand view

Proof

Before (feat-cmux-next): a video a tool writes shows nothing under the call, and a reply link to one is only a path chip. Base gallery, the same transcript without the video variant: /live/?entry=agent-pane.transcript#/agent-pane.transcript/tool-images

After: variant tool-video plays a real 7 s terminal recording of a test run inline, muted on hover, with controls and Expand. Preview: /wt/feat-cmux-next-media-video/?entry=agent-pane.transcript#/agent-pane.transcript/tool-video

šŸ¤– Generated with Claude Code

[!NOTE]
Medium Risk
New host-mediated file access and custom scheme byte-range serving touch WebKit security boundaries; path checks mirror images but large-file streaming and grant lifecycle add new attack surface to validate.

Overview
Adds inline video and audio playback for local files an agent links or tools write in the agent chat, instead of showing only path chips or treating .mp4 paths as images.

The native host exposes a passive media.load {src} bridge call (same path rules as reply images: in-session roots, allowed extensions, up to 4 GB). Approved files get opaque cmux-agent://pane/__media/<token>.<ext> URLs via AgentPaneMediaGrants; the scheme handler serves them with HTTP byte ranges (206/416), rejects symlinks swapped after grant, and adds link.media_refused. Agent pane CSP gains media-src 'self'; remote/https media is refused.

The acpmux UI adds ReplyMedia: lazy media.load near the viewport, inline <video>/<audio> with expand dialog for video, wired from ReplyImage, tool output strips (toolImages + ToolRow), plus styles, i18n, gallery tool-video, and Swift/TS tests.

Reviewed by Cursor Bugbot for commit 9506c26b6cba18486281bccc0add0314c780b55f. Bugbot is set up for automated code reviews on this repo. Configure here.