fix(vllm): pass response_format through omni video EngineInputs

Draft
PR
1mo ago
+22-0

Overview:

Preserve response_format=b64_json on vLLM-Omni video generation (T2V and I2V) so the formatter returns inline base64 instead of a pod-local file:// URL.

This is part A of https://github.com/ai-dynamo/dynamo/issues/14667 only. The independent FFmpeg color-corruption issue (part B) is out of scope.

Upstream PR could not be opened (403). Open against ai-dynamo/dynamo from:

https://github.com/ai-dynamo/dynamo/compare/main...Prudctual:dynamo:cursor/omni-video-response-format-fbb4?expand=1

Details:

_engine_inputs_from_video() constructed EngineInputs without response_format=req.response_format. The output formatter then defaulted to url, so an explicit b64_json request completed with only a worker-local file URL.

Sibling builders were checked for the same omission (#14668 shared root cause):

  • Image (_engine_inputs_from_image): already forwards req.response_format.
  • Audio (AudioGenerationHandler.build_engine_inputs): already maps data_source to EngineInputs.response_format (OpenAI audio uses response_format for the codec).
  • I2V: shares _engine_inputs_from_video, so the one-line assignment covers it.

A unit test now asserts T2V and I2V forward b64_json/url, plus an image regression check.

Where should the reviewer start?

  • components/src/dynamo/vllm/omni/omni_handler.py (_engine_inputs_from_video)
  • components/src/dynamo/vllm/tests/omni/test_omni_handler.py (test_video_and_i2v_forward_response_format)

Related Issues

🔗 This PR is linked to an issue:

Summary

Video omni EngineInputs now carry req.response_format, matching the image path, so b64_json is not dropped.

Validation

  • Sibling builders inspected: image and audio already forward delivery format; I2V uses the patched video builder.
  • Added unit tests in test_omni_handler.py for image, T2V, and I2V response_format forwarding.
  • Source-level check: both image and video EngineInputs constructors pass response_format=req.response_format.
  • Full vllm-omni unit suite is intended to run in CI (pytest markers: unit, vllm, gpu_0, multimodal, pre_merge). This cloud-agent environment does not have vllm-omni installed.
Open in Web Open in Cursor 

Summary by cubic

Fixes vLLM-Omni video generation (T2V and I2V) so response_format=b64_json returns inline base64 instead of a pod-local file URL. _engine_inputs_from_video now forwards req.response_format to EngineInputs, matching the image and audio builders. Adds unit tests verifying b64_json and url forwarding for T2V and I2V. Closes #14667.