feat: overlay targets are scheduling hints, not a serving class
Merged
HA
17h ago+341+341+012345678901234567890123456789-354-354-012345678901234567890123456789
What
A serving class is where a request is routed. Until now, an organisation overlay's targets made the request resolve as a custom class. This PR removes custom: overlay targets become scheduling hints, exactly like account-file priority and targets, and the request keeps its routed class (standard today).
| Before | After |
|---|---|
Overlay targets โ resolved class custom (analytics resolved_serving_class, tariff and cache lookup) | Resolved class is always the routed class (standard) |
| Targets sent through a separate path, even to members that refuse scheduling fields | Targets, priority and account priority all go through one hints path, only to dynamo members that accept scheduling fields |
Overlay class_pricing could price custom | class_pricing prices one of the model's classes (standard). custom is refused by validation, and rows the catalog owns under it are retired on the next apply |
Why
- Classes are destinations.
modelis standard and, later,model:fastis fast. Customer deals (priority, limits, price, tolerations) sit on top of whichever class a request hits. Latency targets are customer labels for alerting and diagnostics; nothing routes on them.customwas left over from the earlier target-routing design. - Fixes a pricing gap. With targets on an overlay, the price lookup used class
customand skipped an account'sstandard-class price, falling back to the model default. The new e2e testoverlay_targets_change_neither_the_class_nor_the_pricecovers it.
Changes
- onwards:
ServingClasshas onlyStandard.ServingTargets, the overlaytargetsfield and the separate stamping path are removed. A stored config that still carries overlaytargetsloads, and the field is ignored.- Hints are not sent to a named-pool member that refuses scheduling fields, matching how its
nvextis already stripped. - The request span records
onwards.serving.priorityandonwards.serving.targets(whether latency targets were sent), separately fromonwards.serving.resolved.
- dwctl:
- The sync maps
model_overlays.targets(ttft_ms,itl_ms,priority) onto the overlay's hints. PricingClassisstandardonly;customjoins the retired class names.- The batch-in-class check that
standardmade redundant is removed.
- The sync maps
- dashboard:
- Pricing lists the model's classes only. A class price is shown winning over the organisation's all-class price, as billing does.
- Overlay details label targets as scheduling hints.
- docs: the overlay how-to, the configuration reference and the class-identity note.
Compatibility
- Existing overlay files keep working:
targets: {ttft_ms, itl_ms, priority}is unchanged on the wire and now applies as hints. - An overlay file that uses
class_pricing: customfails validation (startup and the offline validator). Move such a price tostandard. - No migration.
onwardsdrops public items (ServingTargets,ServingClass::Custom). Its next release is already a major.
Tests
just lint rust,just lint ts: green.just test ts: 805 passed.just test rust: 4244 of 4245 passed. The failure wastest_e2e_ai_proxy_streaming_chat_completions_with_fusillade_header, a known load-timing flake; it passed 3/3 on its own and in the previous full run.- After merging
main(deps/CA bundle): onwards 676 passed, and the affected dwctl suites passed.
๐ค Generated with Claude Code