PR #2605 - harbor-framework/harbor

Summary

Extends ContentPartSchema with an audio type and a corresponding AudioSourceSchema, mirroring the image support added in ATIF-v1.6. A trajectory can now capture speech input, and — as models gain speech output — a spoken agent response, without stepping outside the standard format.

This is deliberately the smallest possible extension: ContentPart.source was already named generically rather than image_source, so audio widens a union instead of adding a parallel mechanism.

{
  "message": [
    { "type": "text", "text": "Answer the question in the recording." },
    {
      "type": "audio",
      "source": {
        "media_type": "audio/wav",
        "path": "audio/step_1_question.wav",
        "duration_sec": 3.2
      }
    }
  ]
}

Media types, and where they come from

Rather than guess, the set is the union of what multimodal and transcription APIs actually accept:

Media type	Evidence
`audio/wav`	Gemini audio understanding; OpenAI audio input; OpenAI transcription
`audio/mpeg`	Gemini (`mp3`); OpenAI transcription (`mp3`, `mpeg`, `mpga`)
`audio/mp4`	OpenAI transcription (`mp4`, `m4a`) — the usual iOS recording output
`audio/aac`	Gemini
`audio/ogg`	Gemini; OpenAI transcription
`audio/flac`	Gemini; OpenAI transcription
`audio/webm`	OpenAI transcription; what a browser's `MediaRecorder` produces, so the form human-recorded task data usually arrives in
`audio/aiff`	Gemini

The audio/mp3 trap. audio/mp3 is not a registered MIME type — the registered one is audio/mpeg — yet it is exactly what the Gemini API documents and returns. A producer copying a provider response verbatim would fail validation against a strict registered-only list. So canonical types are stored, and common alternate spellings are normalized on the way in (audio/mp3, audio/mpga, audio/x-wav, audio/wave, audio/vnd.wave, audio/x-m4a, audio/m4a, audio/x-flac, audio/x-aiff, …). Consumers therefore only ever see one spelling per format.

For reference, Claude has no audio input yet (open feature request on the Messages API), so this is forward-looking there. Worth noting its image list is image/jpeg|png|gif|webp — exactly ATIF's existing ImageSource Literal, which is a good sign the mirroring approach is right.

Design decisions

duration_sec, optional. The only field AudioSource adds over ImageSource. Several providers bill audio per second, dataset statistics need it, and unlike an image's dimensions it cannot be recovered without decoding the file.

A transcript is a separate text part, not a field on the source. Audio is usually paired with text, but expressing that with two content parts uses machinery that already exists and keeps AudioSource purely about locating bytes.

type and source must agree. Widening source to ImageSource | AudioSource would otherwise accept an ImageSource under type="audio" and silently mislabel the content, so ContentPart now rejects a mismatch.

Out of scope, deliberately. Streaming/realtime audio (raw PCM, G.711) is not representable: those encodings are not self-describing, so interpreting them needs sample-rate and channel metadata this schema does not carry. Video has no ATIF content type either. Both are recorded as explicit non-goals in the RFC rather than left ambiguous.

Audio is not a second-class citizen

Adding the model alone would have left audio stored but not usable, so the sites that special-cased images are updated too:

Trajectory validation checks referenced audio files exist, and names audio in the error rather than "image"
has_multimodal_content() and the traces-level multimodal check count audio
Viewer serves audio content types, so clips play inline instead of downloading as octet-stream

One judgement call there: I left .mp4 out of the viewer's map. That container can hold video — this repo already serves OSWorld recording.mp4 as video/mp4 — and mislabelling a video as audio is worse than not mapping it. Audio producers should use .m4a. I also renamed that dict from image_extensions to media_extensions, since it no longer only holds images.

Testing

37 new unit tests: every canonical type, every alias normalization (including case and whitespace), rejection of unknown and video/* types, duration_sec bounds, extra forbidden, the type/source agreement check in both directions, audio in a message and in an observation result, multimodal detection, JSON round-trip, and a guard that ATIF-v1.6/v1.7 trajectories still validate.

Verified the validation path end to end:

missing audio -> False ['trajectory.steps[0].message[1].source.path:
                        referenced audio file does not exist: audio/q.mp3']
present audio -> True  []
url audio     -> True  []   # URLs skipped, not reported missing

uv run pytest tests/unit/ — 5392 passed. The 19 failures on my machine are pre-existing on main (ANSI/terminal-width assertions in CLI output tests) and none are outside that known set. ruff check/format clean; ty check shows only the two pre-existing unresolved imports in traces.py.

Note for reviewers

schema_version becomes "ATIF-v1.8"; v1.0–v1.7 remain valid and a test pins that. Since this changes a published RFC, the media-type list and the alias table are the parts most worth a second opinion — particularly whether normalizing aliases is preferable to rejecting them outright (I favoured interoperability, on the grounds that a producer faithfully copying a provider's mime_type should not be punished).

🤖 Generated with Claude Code