Add support for LLM middleware

Merged
TK
8mo ago
+655-14

why

  • In some scenarios, users need more granular control / access to the LLM at the provider leve

what changed

  • Added support for passing custom "middleware" in the model options

test plan

  • wrote tests
  • tested locally with own middleware

Summary by cubic

Adds first-class LLM middleware to intercept and wrap model calls for logging, usage tracking, or request transforms without changing call sites. Per-call middleware is used only for that call and is not cached.

  • New Features

    • Accepts middleware: LanguageModelV2Middleware in V3 model config; LLMProvider and getAISDKLanguageModel wrap models via wrapLanguageModel from ai.
    • Supports per-call overrides via model options; override middleware is passed through LLMProvider.getClient, applies only to that call, and is never cached.
    • Works across providers and preserves modelId, doGenerate, and doStream. Tests cover usage capture, streaming, errors, chaining, and edge cases.
  • Migration

    • To use: pass middleware in model config or a per-call override, e.g. { modelName: 'openai/gpt-4o', apiKey: '...', middleware }.
    • Middleware is only applied in direct/local execution and is not serialized over HTTP.