fix(server): replay tool history with model-native templates
Merged
GR
1mo ago+1207+1207+0123456789012345678901234567890123456789-156-156-012345678901234567890123456789
Summary
Fix disappearing tool-call history without forcing every model to consume Qwen syntax.
- Normalize OpenAI Chat Completions, Responses, and Anthropic Messages into structured calls: IDs, names, typed arguments, assistant prose, and result associations.
- Render in the selected model's native protocol: Qwen function blocks, Bailing/Laguna argument blocks, Gemma calls/responses, DeepSeek V4 DSML, or DeepSeek V4.1 DSML.
- Preserve call order and retain tool-result IDs/names. Group Responses assistant text and adjacent function-call items into one turn.
- Follow the SGLang/reference DeepSeek V4/V4.1 encoders: reorder result slots by call ID without moving intervening user text, close reasoning before reconstructed calls, and use native result separators. Raw replay remains unchanged.
- Keep exact raw replay for complete built-in tool-memory hits. Reconstruct all calls after misses, eviction, restarts, or missing IDs; preserve malformed and non-object argument strings instead of dropping them.
- Pass structured calls and normalized OpenAI-shaped tool definitions to custom Jinja templates. Bypass raw replay and its canonical-turn cache optimization there: templates such as Gemma's need
tool_callsto find and render subsequent results. - Render Gemma tool definitions and keep calls/results inside the same model turn, including continuation after multiple tool results.
- Update replay documentation and CLI help, including the custom-template field migration.
- Rebase onto
origin/main(c320db9a), preserving the upstream Jinja reasoning-history andpreserve_thinkingsupport. No new dependency or template-system rewrite.
Verification on rebased head 873de718
- CPU-compiled focused runner: 31 passed, 0 failed, 0 skipped. Uses the exact selected test bodies and normalization functions plus the production renderer, tool memory, and bundled Jinja implementation.
- Production-renderer runtime probe: 24 complete prompts match the SGLang DeepSeek V4/V4.1 encoders byte-for-byte — all three APIs, thinking on/off, cold/raw replay, same-name parallel calls, reversed results, and typed nested arguments.
- 6 combined Jinja smoke scenarios preserve structured calls and upstream reasoning history across all three APIs, with
preserve_thinkingdefault/false and remembered raw turns present. - Regression coverage for result ordering across intervening user text, stable missing/unknown-ID fallback, and successive tool turns.
- Full server/GPU suite and live-model inference were not rerun locally on this rebased head. Fresh GitHub CI is tracked separately below/on the checks tab.
Earlier verification (pre-rebase head; author-reported)
- HIP/gfx1201 builds of
luce_serverandtest_server_unit. - Full unit suite: 622 passed, 0 failed, 0 skipped.
- Production-linked runtime probe: all 18 API/native-format combinations, typed nested arguments, parallel calls, and reversed result ordering.
- Official Gemma Jinja output equals an independent Python Jinja oracle for all three APIs, both cold and with remembered raw turns present.
- Official Qwen Jinja conversation output matches an independent Python oracle byte-for-byte; tool definitions are semantically equivalent across the three APIs.
- Real Qwen3-0.6B BF16/HIP server: all three generation APIs return HTTP 200 with both the built-in renderer and official Qwen custom template. Structured histories have the same prompt-token counts as manually serialized native histories. Anthropic
count_tokensagrees with custom-template generation. - Updated
--helpexercised; temporary probes and smoke servers removed.
Regression coverage includes cold/full-hit/partial-hit/evicted/disabled memory, missing IDs, assistant text and null content, empty/malformed/non-object arguments, streaming replay, model-native framing, and Gemma tool-turn continuation.
Limits / tradeoffs
- Live model inference was exercised on Qwen; the other families were verified at the production normalization/rendering layer, not with their model weights.
- Custom Jinja intentionally prioritizes structured-history correctness over byte-identical raw-turn caching.
- Existing bundled-Jinja limitation: filter-mapped
map('upper'), used by the official Gemma template for union-type schemas, is unsupported. Ordinary Gemma schemas passed the canonical-template checks. This PR does not change the Jinja interpreter. - CI for the rebased head must complete. Earlier full-suite/live-model results above apply to the pre-rebase head.
- Non-DeepSeek built-in renderers retain result arrival order; clients should use call order where the native protocol cannot uniquely identify parallel calls.