fix(qwen4exp): prompt prefill cuts only at restore points at least 4096 tokens apart
Merged
DA
7d ago+79+79+01234567890123456789-8-8-0123456789
Since #809, Qwen3.8-Flash-Next ends a prompt forward at every chat boundary, so a prefix-cache hit and a miss see the same cuts. On an agent-shaped prompt (many short messages) that means forwards of about 300 tokens, each paying the full routed-expert read, and cold prefill drops to about a fifth of the single-message rate.
This keeps the guarantee and cuts less often:
ModelBackend::restore_point_spacing()(default 0, next tosnapshot_granularity()): restore points are the first boundary (the system/tools head) plus each boundary at least that many tokens past the last kept one.- One
spaced_restore_points()in the prefix cache.prefix_restore_points()takes the spacing, so every backend that cuts at restore points goes through the same function.reserve_inline_snapplaces its capture at the deepest spaced point. - The rule is prefix-stable: a checkpoint saved at one point is a restore point of every later turn, so a prompt's text still does not depend on what the cache holds.
- Qwen3.8-Flash-Next spaces them 4096 tokens apart. Every other backend keeps every boundary, unchanged.
Results
Strix Halo, UD-Q4_K_XL, main vs this branch:
| main | this branch | |
|---|---|---|
| 50 messages (14,628 tokens), cold prefill | 272 tok/s, 55.1 s | 879 tok/s, 18.2 s |
| same prompt, resent | 1.5 s | 3.1 s (identical text) |
| one 16K message, cold prefill | 1,331 tok/s | 1,306 tok/s |
| one 16K message, resent | 1.5 s | 1.6 s |
A resend re-prefills at most about 4K tokens past the last kept checkpoint, which is the cost of cutting less often.
Validation
test_server_unit647/647, with a new spacing/snap test that derives the expected cut from the prompt's boundaries.- Qwen3.8-Flash-Next prefix-cache model test (lifecycle, partial match, cancel, MTP rollback, park/unpark).
- Quality bench 6/6.
🤖 Generated with Claude Code