fix(thunderagent): inherit backend runtime_config on proxy registration

PR
1mo ago
+534-9

Overview:

ThunderAgent was registering a synthetic public ModelDeploymentCard with a fresh ModelRuntimeConfig. Frontend admission therefore never saw the wrapped backend's runtime contract (token_budget, effective context length, engine capabilities), so over-limit prompts were admitted and then rejected after routing.

This PR copies the backend card's runtime_config (including all runtime_data) onto the proxy card before registration, then overlays CLI parser flags.

Details:

  • After ThunderAgent subscribes to the wrapped worker's MDC stream, wait briefly for a backend card (preferring one that publishes token_budget).
  • Inherit that card's runtime config wholesale: runtime_data via set_engine_specific, plus typed fields, structural-tag settings, and disagg bootstrap metadata.
  • Overlay --dyn-tool-call-parser / --dyn-reasoning-parser / --publish-sglang-generate on top of the inherited contract.
  • Forward the backend kv_cache_block_size on register_model.
  • If no backend card appears, keep previous empty-config registration and log a warning.

This is the smallest fix for the reported admission hole. Removing ThunderAgent's public model registration entirely (the issue's longer-term direction) is left as follow-up; that would require invoking ThunderAgent as a post-preprocess router/flow-control policy.

Upstream compare (403 opening ai-dynamo/dynamo from this token):
https://github.com/ai-dynamo/dynamo/compare/main...Prudctual:dynamo:cursor/thunderagent-proxy-runtime-config-7cc2?expand=1

Where should the reviewer start?

  • components/src/dynamo/thunderagent_router/proxy_card.py — inherit/select/wait helpers
  • components/src/dynamo/thunderagent_router/__main__.py — _proxy_runtime_config() used at register_model
  • components/src/dynamo/thunderagent_router/tests/test_proxy_card.py — token_budget preservation

Related Issues

🔗 This PR is linked to an issue:

Summary

Proxy registration now inherits the backend ModelRuntimeConfig instead of publishing an incomplete synthetic card, so frontend token-budget admission matches the engine.

Validation

  • pytest on components/src/dynamo/thunderagent_router/tests/test_proxy_card.py and test_capacity.py (22 passed; dynamo.llm stubbed in this environment).
  • Overlay test in test_main.py asserts CLI parsers do not drop inherited token_budget.
Open in Web Open in Cursor 

Summary by cubic

Fixes ThunderAgent's public model card so frontend admission sees the backend's runtime contract (token_budget, context length, capabilities) instead of a synthetic empty config. Previously, over-limit prompts were admitted then rejected after routing. Closes #14762.

Bug Fixes

  • Inherits the wrapped worker's runtime config wholesale when registering the proxy model, including runtime_data and structural‑tag settings.
  • Overlays --dyn-tool-call-parser, --dyn-reasoning-parser, and --publish-sglang-generate on top of the inherited contract.
  • Forwards the backend kv_cache_block_size on register_model.
  • Falls back to an empty config with a warning if no backend card appears within the wait window.