PR #2100 - doublewordai/control-layer

Summary

Serving classes become routable. A model that opts in with clay.routing_mode: class_routes serves model from its standard class and model:<class> from each other class it declares in clay.class_routes. Each class has its own upstream (endpoint and upstream model name), realtime prices and default realtime in-flight limit. Accounts can set priority, scheduling hints, toleration presets, limits and prices per class.

Nothing changes for a model until its catalogue entry sets routing_mode: class_routes. Every existing model stays on the legacy route.

What changes

Catalogue (model_provisioning)

New clay.routing_mode (legacy default | class_routes). The provisioning writer sets deployed_models.routing_mode from it, so activation and rollback are catalogue edits.
class_routes requires standard. Other classes are optional (previously standard and fast were both required).
New class_routes.<class>.realtime_inflight_limit. When omitted, the class uses the model's limit.
Class synonyms (aliases) are retired and must be empty. Class names are always model and model:<class>.
Activation is refused for a model without standard or with redirect traffic rules.

Migration (additive): model_serving_classes.realtime_inflight_limit, the realtime_inflight_class_overrides table (account × non-standard class limits), notify triggers, and effective_model_tariff updated to the billing rule below.

Gateway sync

One target per class: model for standard, model:<class> otherwise.
Each target forwards to its class's upstream, without the composite's members, fallback or stream continuation.
Deny rules apply to every class. The model's batch capacity applies to standard only.
The standard target keeps the key model, so existing limiter keys and the batch lane are unchanged.
An activation the sync cannot serve falls back to the legacy route and logs invalid_class_activation.

Account files

classes may name any class the model declares, not only standard.
Limits, prices, priority, hints and toleration presets apply per class. Model-level priority and presets apply to every class.

Requests

Class routes are realtime only. A flex or background request naming model:<class> gets 400 serving_class_realtime_only, and batch lines naming one are refused at upload.
Analytics records model:<class> under the model's name, with requested_serving_class and resolved_serving_class set to the class. The request is priced as that class.
/v1/models lists model:<class> for each non-standard class.
A class route inherits its model's prompt-cache enablement. Standard keeps the model's cache scope.

Docs: model provisioning reference (class routing, account files per class) and the billing precedence guide.

Decisions to review

An account's class-less price covers the standard class only. On another class, the account pays its price for that class if it has one, otherwise the model's public price for that class. A discount on the standard tier therefore never undercuts a premium class. Cache multipliers follow the same rule.
Discovery lists class entries without prices for now. Class prices in /v1/models are a follow-up.
self_hosted_only does not apply to class routes. The class names its upstream directly.
An invalid activation keeps serving the legacy route instead of dropping the model.
Rollback is routing_mode: legacy plus a redeploy. Keep the legacy deployments/routing/tariffs while a model is activated. Older releases reject the routing_mode field, so an image rollback needs the catalogue rolled back first.

Follow-ups (not in this PR)

Class prices in /v1/models.
A class label on existing gateway metrics.
A dry activation of one model against a production-shaped database.
A two-instance Redis check of per-class limits.
A gateway performance re-run.

Testing

Unit and integration tests for catalogue validation, activation and rollback, class limits and prices in account files, per-class targets, toleration presets per class, async/batch refusals, pricing precedence, discovery and continuation.
End to end (test::serving_classes): realtime requests reach each class's upstream and are billed at that class's price; flex and batch naming a class are refused; discovery lists the class.
just lint rust and just test rust pass locally.

🤖 Generated with Claude Code