feat(server): advertise models the way vLLM discovery expects

Merged
FE
3d ago
+556-18

What

Clients that build their model list by asking a vLLM runtime read /v1/models and drop every entry whose owned_by is not vllm or that carries no max_model_len. A gateway that reports its real owners and sets no context field stays invisible to them, so the operator has to write the model list by hand.

This adds models.vllm_discovery_compat (off by default). With it on:

  • GET /v1/models reports owned_by: "vllm" and fills max_model_len from each model's context window.
  • GET /v1/models/{model} can retrieve a single entry.

Claiming to be vLLM is a lie, so it is opt-in per deployment; nothing changes while the flag is off.

Why it is a separate PR

It was the third commit in #1203. @SantiagoDePolonia asked for that PR to be split down to the upstream-header merge, and this feature has no relation to it, so it moves here on top of main.

Verification

  • go test ./config/... ./internal/server/... ./internal/app/...: green, including the new endpoint tests.
  • go vet ./...: the only output is the pre-existing duplicate json:"content" tags in internal/core.
  • Running with the flag on in a live deployment: /v1/models exposes every model with owned_by: "vllm" and a max_model_len, and the vLLM-only consumer lists them.

This change was AI-assisted.

Summary by CodeRabbit

  • New Features
    • Added an opt-in compatibility mode for model discovery clients that expect vLLM-formatted model information. When enabled, model listings and individual model responses identify models as owned by vllm and include max_model_len. Provider-reported limits take precedence over configured context windows; if neither provides a limit, the value is 0. The mode is off by default, and provider-reported limits are not exposed when it is disabled.
  • Documentation
    • Added configuration guidance explaining how to enable the compatibility mode and how model ownership and context limits are reported.