cmux NEXT agent pane: the host reads an attached video's poster, length and key frames

Merged
TE
11d ago
+716-10

Leo, 2026-10-25: "also why dont we have previews of videos we should ... oh attaching a video froze the composer..."

The composer showed a dropped screen recording as a bare file name, and the agent got only the images. The page cannot decode a video: its CSP plays no local file, and reading a recording's bytes in the page is what froze the composer. So the host reads it. This is the native half; the composer chip, the prompt's key frames and the sent bubble each come in their own PRs.

What it does

  • video.info {src} (passive): AVFoundation reads the video off the main actor (AgentPaneVideoInfo.read, @concurrent). It returns the duration, the upright size (after the track's preferred transform), a 320 px JPEG poster and four 1024 px JPEG key frames, taken from the middles of four equal parts. AVAssetImageGenerator (appliesPreferredTrackTransform, 0.5 s tolerance, async image(at:)) decodes only those frames and never reads the whole file. The reply is about 1 MB at most.
  • video.quickLook {src} (spends a gesture): plays the video in Quick Look through the existing replyLinks.quickLookFile.
  • Both accept only a .mov, .mp4 or .m4v (AgentPaneReplyPaths.resolveVideo), in a chat on this Mac, inside the chat's roots, its folder, /tmp or home (the reply image roots; a screen recording lands on the Desktop), never on the deny list (.ssh, keys, .env, also behind a symlink). A chat on another machine is refused: this Mac's files are never read for it.

Why native rather than in-page <video>: the pane's media.load serves only files inside the chat's roots, and Leo's recording is on ~/Desktop. Widening it would grant the page byte-range reads of home. video.info returns only a few JPEGs, and Quick Look plays the file outside the page. The CSP is unchanged.

Tests

AgentPaneVideoInfoTests writes real H.264 movies with AVAssetWriter (red first half, blue second, one rotated a quarter turn). It covers: length, size, a 320×180 poster, four ordered 1024×576 frames (first red, last blue), the rotated video upright, and a non-video .mov and a folder giving nothing. Through the model: a Desktop recording read by path and by file://; refusals for a non-video name, the deny list, outside home and roots, a missing file, a folder and a remote chat; Quick Look only after a click and only for a readable video; the request schema (no extra params) and the op names.

Observed / Not run

  • Fleet (cmux-ci run --class light --script scripts/ci/package-test-lane.sh --arg=suite --arg=Packages/macOS/CmuxNext ..., submitted from cmux16 because the controller refuses huge-blue's tag): at e9ec65ca3af, all 11 suites passed: AgentPaneVideoInfoTests, AgentPaneReplyImageScopeTests, AgentPaneMediaNoAutoActionTests, AgentPaneImageMenuActionsTests, AgentPaneImageActionsTests, AgentPaneMediaDragTests, AgentPaneTerminalFolderTests, AgentPanePickedFolderRootTests, AgentPaneBrowsedFolderRootTests, AgentPaneColdChatRevealTests, SavedEnvironmentsTests (step composer-video-info-green4-e9ec65c…). The reference-movie test failed once on its own precondition at 6ff3b3b5678 (step aec4957d912fdde08a1d3817: a solid-colour movie is smaller than a header). It now proves the pointer's frames are the source's instead.
  • Review (Codex gpt-6-astra, xhigh) fixed along the way: a reference movie could decode another file's media, so the asset now uses AVAssetReferenceRestrictions.forbidAll, with a regression test; path checks and base64 encoding ran on the main actor, so the scope is now read on the main actor and resolved off it, including primaryRoot()'s choice; Quick Look spent its click after an await, so it now spends the click first.
  • Prepush gate clean.
  • Not run: a real-app capture.

No capture: this PR adds two host methods and no UI; nothing on screen changes until the composer chip (#25181) calls them, and that PR carries the gallery states.

Changelog

none (the chip PR carries the user-visible entry)

risk: normal (a passive read-only host method behind the same path policy as reply images, plus a gesture-gated Quick Look)

Agent session: huge-blue

🤖 Generated with Claude Code