Show the images and PDFs an agent's tools return or write inline in the chat

Merged
TE
1mo ago
+410-19

Summary

Image outputs from an agent's tool calls now show inline in the chat, and a PDF an agent writes or links shows as a thumbnail of its first page. Before this, the transcript dropped returned images and showed a PDF only as a file chip.

This PR now includes #18357 (closed into this one), so it lands both changes:

  • Returned images (from #18357): a tool that returns an image, such as a browser screenshot, shows it. ACP image content blocks are kept as data URLs, png/jpeg/gif/webp only, up to 8 MB each.

  • Written images (from #18357): a tool that writes or prints an image file shows that file: the edited file, the call's locations, and absolute image paths in its output. They load through the host like a reply's local image (image.load), so only files inside the session's folders show. At most four per call, each capped at 360x220 and decoded lazily. A click opens the image viewer (#18200), as do reply images, through the shared OpenableImage.

  • Host-drawn page: the host draws page 1 of the PDF with CoreGraphics, on white, with the longest side at 1024 px. It hands the pane a PNG data URL through the existing image.load path. The page never receives the document and keeps its img-src data: CSP.

  • Limits: only page 1 is decoded, so PDFs up to 100 MB qualify (other images stay at 10 MB). A file that CoreGraphics can't read as a PDF is refused, as is a locked one. As for any reply image, only files inside the session's folders load.

  • Tool outputs: a .pdf a tool edits, lists in its locations or prints as an absolute path now joins the tool's image strip. A reply link to a local PDF shows the same thumbnail through ReplyImage.

  • Click: opens the in-pane image viewer (#18200), never a browser tab.

Targets feat-cmux-next and carries #18357's commits. Part of bead cx-ncc.51. Video and artifact links follow in separate PRs.

Testing

  • #18357's part: red commit 5c3dbb8ac6b, then the fix; toolImages.test.tsx covers returned and written images.
  • Red commit 788b6627c3f, then the fix.
  • bun test src/agent-session/acpmux/conversation/toolImages.test.tsx (local): 3 pass, 1 fail at the red commit; 4 pass after the fix.
  • AgentPaneReplyImageTests.aPDFShowsItsFirstPageAsAPNG checks a two-page portrait PDF. The result must be a PNG data URL whose longest side is thumbnailSide and which stays portrait, and a fake .pdf must be refused. Fleet Swift suite (step 18909ba90b44, run-swift-testing-suites.sh at c24deb96ef7, which carries this PR's Swift unchanged): passes.
  • Review follow-up: a page smaller than the thumbnail was drawn at its natural size in white margins, because CoreGraphics' page transform never scales up. Red commit 6c94740777b samples pixels on a Letter page and adds a /Rotate 90 page. Fleet suite at 6c94740777b: aPDFShowsItsFirstPageAsAPNG fails with 2 issues, one on each page. The fix 8eb55249bfb builds the transform by hand and clips to the crop box. At 8eb55249bfb the rotated page passes, but the Letter-page check still failed: that page stores its blue as sRGB, and its device-RGB value missed the exact-byte threshold. 476b12d7583 matches blue by hue, and the test passes on the fleet at 476b12d7583.
  • vp check and vp fmt pass on the changed files. verify-local.py --only swift-syntax passes on the changed Swift file. The agent pane bundle is no longer committed (base dropped it); builds generate it.

Changelog

Added: Images an agent's tools return or save (screenshots, charts) show inline in the agent chat, and a click zooms them
Added: A PDF an agent writes or links shows a first-page thumbnail in the agent chat, and a click zooms it

Proof

Before (feat-cmux-next): a tool call shows only text; returned and written images are dropped, and a PDF shows only as a file chip. The base gallery has no tool-images variant: /live/?entry=agent-pane.transcript

After: variant tool-images shows a sign-in page screenshot a browser tool returned, a build-time chart a script saved, and a three-page build-report PDF a script wrote, shown as its real first page. Preview: /wt/feat-cmux-next-media-thumbnails/?entry=agent-pane.transcript#/agent-pane.transcript/tool-images

The gallery host's branch sync is stuck on a deleted branch, so the preview shows the newest build before the latest base merge. The variant's content is the same.

šŸ¤– Generated with Claude Code

[!NOTE]
Medium Risk
Adds host-side PDF decoding and new transcript UI for tool outputs; bounded by size caps and session-folder rules, but CoreGraphics PDF handling is a new attack surface for malicious files.

Overview
Tool calls in the agent transcript now show inline image thumbnails under each row (even when collapsed): ACP image content blocks are kept as capped data URLs, and absolute image/PDF paths from edits, locations, or output text load through the existing host image.load path via ReplyImage, at most four per call.

On macOS, local PDFs (replies and tool outputs) no longer pass through the raster pipelineā€”the host renders page 1 with CoreGraphics to a PNG data URL (1024px longest side, up to 100 MB), including /Rotate, so the webview still only sees data: images under CSP.

OpenableImage centralizes click-to-zoom for shown images (markdown inline, reply images, and tool strips); gallery adds a tool-images transcript variant and tests cover PDF thumbnails and toolImages merging.

Reviewed by Cursor Bugbot for commit 654801feccac2e4c3eb37a89e218ed5f3f395ac3. Bugbot is set up for automated code reviews on this repo. Configure here.