PR #30921 - manaflow-ai/cmux

Leo, 2026-11-01, after he attached a 16 s mp4 in a cmux NEXT chat and the agent had to hand-write ffmpeg -vf fps=1,scale=640:-1,tile=4x4 into /tmp to look at it: "i feel like cmux-next should give u better tools to deal with this kind of thing ... you can see what we can work on", then "can you help work on all these".

What

Two new tools on the cmux MCP server (cmux-tui cli/mcp/video_tools.rs), listed in cmux_exec's common tools so agents pick them over ffmpeg:

video_info {path}: duration (ms), width and height as shown (after rotation), fps, has audio.
video_frames {path, at | range | scenes, count, maxSide}: frames as MCP image blocks, plus one text block with the duration, size, fps, audio and each frame's time.
- at: [seconds…]: exact frames at those times (1 to 16).
- range: {from, to, count}: count frames evenly from from to to, both included (defaults 0, the end, 8).
- scenes: true (with count, default 8): the first frame plus the moments the picture changes most, in time order, each with its change score.
- neither: count frames at the middles of equal parts of the whole video (default 8).
- maxSide: longest side in px, 64 to 2048, default 1280; never upscaled.

Bounded output: at most 16 frames a call and 6 MiB of base64 JPEG. A call over the budget returns the first frames and says how many it left out, and what to ask for next. cmux_exec's per-script image cap goes from 8 to 16 (video_tools::MAX_FRAMES) so a 16-frame call fits; its 8 MiB byte cap is unchanged.

How

Rust (the logic): validates arguments, resolves the path (absolute, ~/, or relative to the chat's folder, as open does), picks the times and the decode tolerance (exact for at, a quarter of the spacing otherwise), finds scene changes from 64 small grey samples (mean absolute difference between neighbours, a 1.5-grey-level noise floor), and builds the image blocks. The result has no structuredContent, so cmux_exec moves the images into its own result (code_mode::script_answer).
Swift (the decoder): an app control method video.frames (CmuxNextApp/Control/VideoControl.swift) over a new AgentVideoFrames (CmuxNextAgentPane/AgentVideoFrames.swift). It uses the composer's AVFoundation decoder and its helpers from AgentPaneVideoInfo (isRegularFile, jpeg, and forbidAll reference restrictions). It decodes only the asked frames, never the whole file.
Same file, same machine: the server sends the file's device and inode as it sees them. The app refuses with not_found if its own stat of the path differs, so a chat whose MCP server runs on another machine never gets frames from a same-named file on the Mac.

Remote and Cloud sessions

The decode runs in the cmux app on the Mac. In a Cloud or ssh chat the MCP server runs on the remote machine, and the call fails cleanly in one of two ways. If no app answers, the error is app.not_found or app.unreachable. If the path belongs to another machine, the app answers not_found and the message tells the agent to use that machine's own tools. Follow-up: a decoder on the remote side (an ffmpeg/ffprobe backend in the same Rust module, picked when no local app owns the file) gives remote chats the same tool with the same schema.

Overlap

Another lane is changing AgentPaneVideoInfo.swift's frame picking (scene detection, timestamps). This PR does not touch that file. It only calls its existing isRegularFile and jpeg helpers from a new file. Scene picking for this tool lives in Rust (video_tools::scene_times), since base has no Swift scene picker. If the other lane's Swift picker lands, the tool can keep its own or switch to it later; nothing conflicts.

Tests

Rust, video_tools::tests: schema and caps, argument errors, time picking and clamping, scene picking, image blocks after one text block, the probe-then-frames request shape (path, dev/ino, side, tolerance, budget), the scene two-pass flow, left-out frames, a refusal from the app, and a missing file.
Swift, AgentVideoFramesTests: numbers only, frames at exact times and size (red then blue, never upscaled), grey samples that change with the picture, the byte budget, and a non-video file.

Risk: additive. Two new tools, one new read-only app method, and one constant raised (8 to 16 images per cmux_exec script, under the same 8 MiB cap).

🤖 Generated with Claude Code

Agent session: agent session

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Summary by cubic

Adds video_info and video_frames tools to the cmux MCP server so agents can inspect a video directly instead of hand-writing ffmpeg commands.

video_info {path} returns duration (ms), displayed width and height, fps, and audio presence.
video_frames {path, at | range | scenes, count, maxSide} returns frames as MCP image blocks plus a text block listing each frame's time; capped at 16 frames and 6 MiB of base64 JPEG. An over-budget call returns the first frames and says how many it left out, or tells the agent to lower maxSide when the first frame alone is too big.
Frame times, sizes, byte budget, and scene changes are picked by the Rust server, with scene sampling starting from the first frame; the macOS app decodes only the asked frames via AVFoundation.
The app verifies the file's device and inode, so a chat whose MCP server runs on another machine fails cleanly instead of reading a same-named file on the Mac.
Raises cmux_exec's per-script image cap from 8 to 16 so a 16-frame call fits; the 8 MiB byte cap is unchanged.