fix(qwen4exp): prompt prefill cuts only at restore points at least 4096 tokens apart

Merged
DA
7d ago
+79-8

Since #809, Qwen3.8-Flash-Next ends a prompt forward at every chat boundary, so a prefix-cache hit and a miss see the same cuts. On an agent-shaped prompt (many short messages) that means forwards of about 300 tokens, each paying the full routed-expert read, and cold prefill drops to about a fifth of the single-message rate.

This keeps the guarantee and cuts less often:

  • ModelBackend::restore_point_spacing() (default 0, next to snapshot_granularity()): restore points are the first boundary (the system/tools head) plus each boundary at least that many tokens past the last kept one.
  • One spaced_restore_points() in the prefix cache. prefix_restore_points() takes the spacing, so every backend that cuts at restore points goes through the same function. reserve_inline_snap places its capture at the deepest spaced point.
  • The rule is prefix-stable: a checkpoint saved at one point is a restore point of every later turn, so a prompt's text still does not depend on what the cache holds.
  • Qwen3.8-Flash-Next spaces them 4096 tokens apart. Every other backend keeps every boundary, unchanged.

Results

Strix Halo, UD-Q4_K_XL, main vs this branch:

mainthis branch
50 messages (14,628 tokens), cold prefill272 tok/s, 55.1 s879 tok/s, 18.2 s
same prompt, resent1.5 s3.1 s (identical text)
one 16K message, cold prefill1,331 tok/s1,306 tok/s
one 16K message, resent1.5 s1.6 s

A resend re-prefills at most about 4K tokens past the last kept checkpoint, which is the cost of cutting less often.

Validation

  • test_server_unit 647/647, with a new spacing/snap test that derives the expected cut from the prompt's boundaries.
  • Qwen3.8-Flash-Next prefix-cache model test (lifecycle, partial match, cancel, MTP rollback, park/unpark).
  • Quality bench 6/6.

🤖 Generated with Claude Code