fix(decopilot): show MCP image results to the model instead of base64 text

YU
11d ago
+154-1

What is this contribution about?

When an MCP tool returns image content, decopilot sends it to the model as text. toModelOutput in mcp-tools.ts runs JSON.stringify on every non-text block, so the model gets a JSON string holding the base64 bytes.

Measured with the new Figma MCP (decocms/mcps#558). One get_design_context call on a 450×768 frame returned 100,341 bytes, about 25k tokens, almost all of it the PNG as base64. The model cannot read that. On larger frames the total goes over MAX_RESULT_TOKENS (32k), and truncateForModel then replaces the whole result, including the useful text outline, with a read_tool_output preview.

Studio already has a way to show tool images to the model. take_screenshot and the VM Read tool put images on pendingImages, and prepareStep attaches them as a user message. The comment in built-in-tools/index.ts explains why: images inside tool-result messages are not supported by every provider, but images in user content are. This PR routes MCP images through the same queue:

  • attachToolResultImages swaps each image block for a short placeholder, [image/png image, 98 KB, attached below], and queues the bytes on pendingImages.
  • If there is no queue, or the image is over 5 MB (Anthropic's limit), the image is only described and nothing is attached.
  • The truncation check now runs on the result without the base64, so the text is no longer dropped because of the image.
  • Each toolCallId queues its images only once, even if toModelOutput runs again.
  • Only the model-bound copy changes. The tool result streamed to the UI is untouched.
  • tools.ts passes extras.pendingImages, the queue the built-in tools already share, into toolsFromMCP.

Behavior change on non-image results: none. They take the same code path as before.

Why not put the image inside the tool result?

AI SDK v7 does support { type: "content", value: [..., { type: "file", mediaType, data }] }. I checked how each provider converts it:

ProviderImage in a tool result
Anthropicnative image block
OpenAI Responsesinput_image
Googlefunction-response parts
OpenRouterimage part
OpenAI Chat CompletionsJSON.stringify: back to base64 text

The pendingImages path works the same way for all of them, and it is the convention Studio already uses.

Related PRs

  • #5396 caps screenshot size at Anthropic's 8000 px limit. The Figma MCP already renders under that.
  • #7021 strips inline base64 image bytes when thread parts are persisted. Images are therefore not replayed on later turns, and that stays true here.

How did you verify your code works?

  • New mcp-tools.test.ts (3 tests):
    • an image is queued with a data URL and label, and the result keeps a placeholder;
    • with no queue, or above the size cap, the image is only described and nothing is queued;
    • end to end through toolsFromMCP: a result of about 400 KB of base64 plus text gives the model "[image/png image, 293 KB, attached below]\nFRAME Card 320x200" instead of a truncation preview. The image is queued once across two toModelOutput calls, and the original result keeps its image block.
  • bun test src/harnesses/lib/decopilot: 188 pass, 0 fail.
  • tsc --noEmit in apps/api: clean.
  • biome format and oxlint on the changed files: clean. The kebab-case filename warning already existed.

How to Test

  1. Connect an MCP that returns images, for example the Figma MCP from decocms/mcps#558, or any MCP with a screenshot tool.
  2. In a decopilot chat, ask it to call the tool. With Figma: get_design_context on a frame link.
  3. Expected: the model describes what is in the image, the text part of the result is intact, and no base64 shows up in the model input. The tool card in the UI looks the same as before.

Review Checklist

  • PR title is clear and descriptive
  • Changes are tested and working
  • Documentation is updated (if needed)
  • No breaking changes

🤖 Generated with Claude Code


Summary by cubic

Shows MCP tool image results to the model as attached images instead of unreadable base64 text.

  • MCP image content previously reached the model as a stringified JSON blob of base64 bytes, costing roughly 1 token per 4 bytes and often pushing results past the 32k-token cap so even the text was replaced by a read_tool_output preview.
  • Routes MCP images through the existing pendingImages queue used by take_screenshot and the VM Read tool, leaving a short text placeholder in the model-bound tool result.
  • Images over 5 MB, or when no queue is available, are only described and not attached.
  • Only the model-bound copy changes; the tool result streamed to the UI keeps its original image block.
  • Non-image results take the same code path as before.