Add support for LLM middleware
Merged
TK
8mo ago+655+655+012345678901234567890123456789-14-14-01234567890123456789
why
- In some scenarios, users need more granular control / access to the LLM at the provider leve
what changed
- Added support for passing custom "middleware" in the model options
test plan
- wrote tests
- tested locally with own middleware
Summary by cubic
Adds first-class LLM middleware to intercept and wrap model calls for logging, usage tracking, or request transforms without changing call sites. Per-call middleware is used only for that call and is not cached.
-
New Features
- Accepts
middleware: LanguageModelV2MiddlewareinV3model config;LLMProviderandgetAISDKLanguageModelwrap models viawrapLanguageModelfromai. - Supports per-call overrides via
modeloptions; override middleware is passed throughLLMProvider.getClient, applies only to that call, and is never cached. - Works across providers and preserves
modelId,doGenerate, anddoStream. Tests cover usage capture, streaming, errors, chaining, and edge cases.
- Accepts
-
Migration
- To use: pass
middlewareinmodelconfig or a per-call override, e.g.{ modelName: 'openai/gpt-4o', apiKey: '...', middleware }. - Middleware is only applied in direct/local execution and is not serialized over HTTP.
- To use: pass