fix(server): replay tool history with model-native templates

Merged
GR
1mo ago
+1207-156

Summary

Fix disappearing tool-call history without forcing every model to consume Qwen syntax.

  • Normalize OpenAI Chat Completions, Responses, and Anthropic Messages into structured calls: IDs, names, typed arguments, assistant prose, and result associations.
  • Render in the selected model's native protocol: Qwen function blocks, Bailing/Laguna argument blocks, Gemma calls/responses, DeepSeek V4 DSML, or DeepSeek V4.1 DSML.
  • Preserve call order and retain tool-result IDs/names. Group Responses assistant text and adjacent function-call items into one turn.
  • Follow the SGLang/reference DeepSeek V4/V4.1 encoders: reorder result slots by call ID without moving intervening user text, close reasoning before reconstructed calls, and use native result separators. Raw replay remains unchanged.
  • Keep exact raw replay for complete built-in tool-memory hits. Reconstruct all calls after misses, eviction, restarts, or missing IDs; preserve malformed and non-object argument strings instead of dropping them.
  • Pass structured calls and normalized OpenAI-shaped tool definitions to custom Jinja templates. Bypass raw replay and its canonical-turn cache optimization there: templates such as Gemma's need tool_calls to find and render subsequent results.
  • Render Gemma tool definitions and keep calls/results inside the same model turn, including continuation after multiple tool results.
  • Update replay documentation and CLI help, including the custom-template field migration.
  • Rebase onto origin/main (c320db9a), preserving the upstream Jinja reasoning-history and preserve_thinking support. No new dependency or template-system rewrite.

Verification on rebased head 873de718

  • CPU-compiled focused runner: 31 passed, 0 failed, 0 skipped. Uses the exact selected test bodies and normalization functions plus the production renderer, tool memory, and bundled Jinja implementation.
  • Production-renderer runtime probe: 24 complete prompts match the SGLang DeepSeek V4/V4.1 encoders byte-for-byte — all three APIs, thinking on/off, cold/raw replay, same-name parallel calls, reversed results, and typed nested arguments.
  • 6 combined Jinja smoke scenarios preserve structured calls and upstream reasoning history across all three APIs, with preserve_thinking default/false and remembered raw turns present.
  • Regression coverage for result ordering across intervening user text, stable missing/unknown-ID fallback, and successive tool turns.
  • Full server/GPU suite and live-model inference were not rerun locally on this rebased head. Fresh GitHub CI is tracked separately below/on the checks tab.

Earlier verification (pre-rebase head; author-reported)

  • HIP/gfx1201 builds of luce_server and test_server_unit.
  • Full unit suite: 622 passed, 0 failed, 0 skipped.
  • Production-linked runtime probe: all 18 API/native-format combinations, typed nested arguments, parallel calls, and reversed result ordering.
  • Official Gemma Jinja output equals an independent Python Jinja oracle for all three APIs, both cold and with remembered raw turns present.
  • Official Qwen Jinja conversation output matches an independent Python oracle byte-for-byte; tool definitions are semantically equivalent across the three APIs.
  • Real Qwen3-0.6B BF16/HIP server: all three generation APIs return HTTP 200 with both the built-in renderer and official Qwen custom template. Structured histories have the same prompt-token counts as manually serialized native histories. Anthropic count_tokens agrees with custom-template generation.
  • Updated --help exercised; temporary probes and smoke servers removed.

Regression coverage includes cold/full-hit/partial-hit/evicted/disabled memory, missing IDs, assistant text and null content, empty/malformed/non-object arguments, streaming replay, model-native framing, and Gemma tool-turn continuation.

Limits / tradeoffs

  • Live model inference was exercised on Qwen; the other families were verified at the production normalization/rendering layer, not with their model weights.
  • Custom Jinja intentionally prioritizes structured-history correctness over byte-identical raw-turn caching.
  • Existing bundled-Jinja limitation: filter-mapped map('upper'), used by the official Gemma template for union-type schemas, is unsupported. Ordinary Gemma schemas passed the canonical-template checks. This PR does not change the Jinja interpreter.
  • CI for the rebased head must complete. Earlier full-suite/live-model results above apply to the pre-rebase head.
  • Non-DeepSeek built-in renderers retain result arrival order; clients should use call order where the native protocol cannot uniquely identify parallel calls.