feat(server): advertise models the way vLLM discovery expects
Merged
FE
3d ago+556+556+012345678901234567890123456789-18-18-01234567890123456789
What
Clients that build their model list by asking a vLLM runtime read /v1/models and drop every entry whose owned_by is not vllm or that carries no max_model_len. A gateway that reports its real owners and sets no context field stays invisible to them, so the operator has to write the model list by hand.
This adds models.vllm_discovery_compat (off by default). With it on:
GET /v1/modelsreportsowned_by: "vllm"and fillsmax_model_lenfrom each model's context window.GET /v1/models/{model}can retrieve a single entry.
Claiming to be vLLM is a lie, so it is opt-in per deployment; nothing changes while the flag is off.
Why it is a separate PR
It was the third commit in #1203. @SantiagoDePolonia asked for that PR to be split down to the upstream-header merge, and this feature has no relation to it, so it moves here on top of main.
Verification
go test ./config/... ./internal/server/... ./internal/app/...: green, including the new endpoint tests.go vet ./...: the only output is the pre-existing duplicatejson:"content"tags ininternal/core.- Running with the flag on in a live deployment:
/v1/modelsexposes every model withowned_by: "vllm"and amax_model_len, and the vLLM-only consumer lists them.
This change was AI-assisted.
Summary by CodeRabbit
- New Features
- Added an opt-in compatibility mode for model discovery clients that expect vLLM-formatted model information. When enabled, model listings and individual model responses identify models as owned by
vllmand includemax_model_len. Provider-reported limits take precedence over configured context windows; if neither provides a limit, the value is0. The mode is off by default, and provider-reported limits are not exposed when it is disabled.
- Added an opt-in compatibility mode for model discovery clients that expect vLLM-formatted model information. When enabled, model listings and individual model responses identify models as owned by
- Documentation
- Added configuration guidance explaining how to enable the compatibility mode and how model ownership and context limits are reported.