Add instrumentation version 4 to match OTel GenAI semantic conventions for multimodal input

Merged
AD
9mo ago
+570-23

Adds support for OpenTelemetry GenAI semantic conventions for multimodal content via a new instrumentation version 4. This addresses the issue where image, audio, video, and document content was not following the official spec format.

Changes:

  • Add UriPart and BlobPart TypedDicts to _otel_messages.py for the spec-compliant formats
  • Add version 4 to InstrumentationSettings (versions 1-3 remain unchanged)
  • Update messages.py to convert media URLs and binary content to the new format when version >= 4
  • Apply version 4 format to both input messages (UserPromptPart) and output messages (ModelResponse/FilePart)
  • Add comprehensive tests for version 4 multimodal inputs and outputs
  • Update logfire.md documentation to explain version 4

Version 4 changes the format in gen_ai.input.messages and gen_ai.output.messages:

URL-based media (ImageUrl, AudioUrl, DocumentUrl, VideoUrl):

  • Old (v1-3): {"type": "image-url", "url": "..."}
  • New (v4): {"type": "uri", "modality": "image", "uri": "...", "mime_type": "..."}

Binary content (BinaryContent in input, FilePart in output):

  • Old (v1-3): {"type": "binary", "media_type": "...", "content": "..."}
  • New (v4): {"type": "blob", "modality": "image", "mime_type": "...", "content": "..."}

Privacy controls:

  • mime_type is always included (metadata)
  • uri and content are omitted when include_content=False (sensitive data)

This is a non-breaking change as version 2 remains the default.

Fixes #4200