Preserve tool provenance for multimodal return content

Closed
DS
2mo ago
+1581-612

This pull request was posted by Codex Desktop using gpt-5.6-sol on behalf of David.
This description was last updated by Claude Code using claude-opus-5 on behalf of David.

Why we make these changes

Multimodal tool output is moved into ordinary user-prompt content by providers whose native tool-result shape cannot carry it. That loses the originating tool name and call ID in stored history and makes replayed tool output indistinguishable from a real user upload.

This adds model-neutral provenance to the stored part, then renders either a verified native tool-result block or a fixed fallback marker only when building the provider request. The marker is applied to media alone: text passed through ToolReturn.content is already attributed by the tool result it accompanies, so its prompt stays byte-identical to today on every provider. It also makes our Cohere mapper reject tool media it cannot carry instead of silently dropping it.

Breaking change: multimodal tool returns on Cohere now raise

Affects Cohere users whose tools return an image, document, audio, or video. Nothing else about Cohere changes, and no other provider changes.

Symptom. Code that ran without error before now raises on the run after upgrading:

UserError: The Cohere integration does not yet support multi-modal content in tool returns.

Cause. That media was never reaching Cohere. Our mapper sent the tool's text and dropped the attachment, so the model answered about a file it had never been given. The silence was the bug (#6662); raising is the fix.

What to do. Our Cohere integration can carry media in neither tool results nor user prompts today, so on Cohere have the tool return text instead: a description, an extracted transcript, or a URL. If the model needs to see the file itself, use a provider whose mapper carries that file type. Cohere media support is tracked in #7646.

New public surface

  • ToolReturnProvenance
  • UserPromptPart.source
  • MistralModelProfile, with mistral_supports_media_in_tool_returns (default True)
Goal: preserve provenance in stored and replayed history

Before → after

UserPromptPart(content=[image])  # -> origin lost
UserPromptPart(content=[image], source=ToolReturnProvenance(tool_name='get_image', tool_call_id='call_1'))
Playground 
from pydantic_ai import ToolReturnProvenance
from pydantic_ai.messages import ModelMessagesTypeAdapter, ModelRequest, UserPromptPart

messages = [
    ModelRequest(
        parts=[
            UserPromptPart(
                content='tool output',
                source=ToolReturnProvenance(tool_name='lookup', tool_call_id='call_1'),
            )
        ]
    )
]
assert ModelMessagesTypeAdapter.validate_json(ModelMessagesTypeAdapter.dump_json(messages)) == messages

Tests: serialization, cross-provider replay, and realtime history.

What changes for existing users: tool-produced prompt content gains persisted provenance; ordinary user-prompt JSON remains unchanged.

Goal: render provider-native media or an explicit fallback

Before → after

['This is file abc:', image]  # -> ordinary user content
['<pydantic_ai:tool_return tool_name="lookup" tool_call_id="call_1" />', 'This is file abc:', image]

'plain tool text'  # -> unchanged; text needs no marker

Media stays in the tool result where the provider's tool-result API accepts it (Mistral behind its new profile flag, and profile-supported Google and Bedrock media); everywhere else it travels as prompt content behind the marker. Where our mapper can carry the media neither way, it raises rather than dropping it.

Playground 
from pydantic_ai import Agent, BinaryContent

agent = Agent('openai:gpt-5-mini')

@agent.tool_plain
def screenshot() -> BinaryContent:
    return BinaryContent(data=b'png', media_type='image/png')

result = agent.run_sync('Call screenshot.')

Test: centralized provider request/VCR matrix, with exact mapper snapshots for provider fallbacks.

What changes for existing users: tool-produced media that spills into prompt content is now attributable to its tool call, and media our Cohere mapper cannot carry raises instead of vanishing. Text-only ToolReturn.content is untouched, so a user not returning media from tools sees no prompt change on any provider.

Checklist

  • Any AI generated code has been reviewed line-by-line by the human PR author, who stands by it.
  • No breaking changes in accordance with the version policy.
  • PR title is fit for the release changelog.