Round-trip multimodal tool returns through the AG-UI and Vercel AI adapters (history and streaming)

Merged
DS
4mo ago
+1037-115

Summary

Makes multimodal tool-return content — BinaryContent, ImageUrl, DocumentUrl, and the other file types, including files nested inside a dict/list — survive the trip through the AG-UI and Vercel AI adapters as the original file types, instead of being flattened to text.

This matters most for a streaming frontend: the message history is rebuilt from the stream events, so a file a tool produces during a run has to be carried in the tool-result event itself, or it's already lost by the next step — the model would see a text placeholder rather than the image, breaking multi-step reuse and prompt-cache prefix preservation. So both the live event stream and history serialization (dump_messages/load_messages) now carry the full content, using one canonical serialization each way.

How

Each adapter serializes a tool return's full content into its protocol's tool-output slot on the way out, and rehydrates it via the ToolReturnContent discriminator on the way back:

  • AG-UI: ToolMessage.content (history) and ToolCallResultEvent.content (stream) both carry the content via the shared dump_tool_return_content — AG-UI already carries structured tool returns as JSON there, so file-bearing returns use the same channel (base64 for binary, URL for file URLs), round-tripping through any frontend with no sidecar and no flag.
  • Vercel AI: the tool output field (history) and the tool-output-available chunk (stream) both carry the content via the shared tool_return_output.
  • Rehydration uses the ToolReturnContent discriminator; image BinaryContent is narrowed to BinaryImage. A plain-text return stays a plain string.

Files from client-side tools (Vercel)

A Vercel AI client-side tool (resolved server-side as an external/deferred tool call) can return a file by putting a { kind: 'binary', media_type, data } shape in its output. The data accepts a base64 string, or the raw byte shapes a JS frontend produces from a Uint8Array / Node Buffer via JSON.stringify — all normalized to bytes at the wire boundary, so callers needn't base64-encode by hand.

Trade-off

A file in a tool result is now a base64/JSON value on the wire rather than a human-readable placeholder like [File: image/jpeg]. This is intentional: tool results are rarely rendered directly to the user, round-trip fidelity matters more here, and a file returned by a tool can't be rendered nicely by a generic frontend anyway. Neither protocol has a native "this tool result is an image, render it" type today (AG-UI tool-result content is str, Vercel's is Any); the AG-UI gap may be worth raising upstream.

Relationship to the split-out PRs

The serialization groundwork was extracted and merged separately; this PR builds on all of it:

  • #6191 (merged) — ToolReturnContent discriminator + graceful passthrough hardening.
  • #6193 (merged) — single-sourced BinaryImage narrowing.
  • #6232 (merged) — split preserve_file_data into allow_uploaded_files (inbound security) and the AG-UI-only preserve_file_data (the FilePart/UploadedFile sidecar round-trip). This PR uses allow_uploaded_files for inbound uploaded-file handling; multimodal tool-return files are always preserved outbound (no flag).
  • #6170 (closed) — its premise (stripping outbound tool-return files as a security measure) was a non-problem.

Tests

Streaming and history round-trip tests for both adapters, including files nested in a dict/list, the client-side-tool byte shapes, and explicit stream → replay-as-history → rehydrate tests proving a streamed file is recovered as its original type. test_ui, test_ag_ui, test_vercel_ai green; pyright + coverage clean.

Closes #5811.
Closes #5147.

🤖 Generated with Claude Code