fix(server): each turn reuses the previous one, long prompts share time, late system messages work
Fixes for problems found by serving the Manage models the way Manage does (Qwen3.8-Flash-Next split over an R9700 + Strix Halo, Qwen 3.8 27B on the R9700, DeepSeek V4.1 Flash on an R9700 + 192 GB Strix Halo) to scripted chats and agents and to real Claude Code and Codex sessions.
What was wrong
| What a person saw | Models | Cause |
|---|---|---|
| A follow-up question on a long document re-read the whole document | all three: 103 s (Flash-Next, 130K tokens), 532 s (Qwen 27B, 118K), 72 s (V4.1, 33K) | The prefix cache saved a turn before its last message, so the message the follow-up extends was never saved. |
| Every reply in a chat re-read the whole conversation; an agent re-read up to 4K tokens per turn | Flash-Next | Restore points 4096 tokens apart (#828) moved every save down to the last of them. |
| Agents running at once rarely reused their own turns | Flash-Next, Qwen 27B | The prefix cache held one capture at a time, so requests prefilling beside it saved nothing. |
| A person chatting beside a long prompt got 1.6 tok/s (Flash-Next) or 0.6 tok/s (Qwen 27B) | Flash-Next, Qwen 27B | Each step paired a decode token with a prompt chunk; on Qwen 27B such a step also runs without the drafter. |
| A long document sent to Qwen 27B on its own crashed the server 79 s in | Qwen 27B | Paged attention asked for more partials than the R9700 had left (ROCm error: out of memory). |
| An app that sends a system message after the first user message failed on every request (HTTP 500); Claude Code 2.1 does this and retried 10 times | Flash-Next | The model's template rejects a late system message. |
Anthropic API clients asking for adaptive thinking (Claude Code does) got none; replies in every app started with blank lines; OpenAI API clients (Codex, for one) showed 0 cached tokens | all | adaptive thinking turned thinking off; the newlines after </think> were sent as text; usage had no cached-token field. |
Changes
- fix(server): the prefix cache keeps each turn's end, for every request at once. Each prefilling request holds its own capture reservation, and concurrent serving prunes the snapshots a new capture supersedes, as single-sequence serving does.
- feat(server): requests decoding beside a long prompt keep half the time (
StepPlanLimits::prefill_time_share), and a prompt that fits in one step runs first. - fix(server): a late system message no longer fails a request, replies start without blank lines. For any app: a template that rejects a late system or developer message gets it as a user turn, in place;
adaptivethinking means the model's default; a template error answers 400. - feat(server): usage reports the prompt tokens the prefix cache served (
prompt_tokens_details.cached_tokens,input_tokens_details.cached_tokens). - fix(server): a late system message stays one where the template takes it.
- fix(server): every model keeps each turn's end, and the turn before it for edits.
- Every backend saves the prompt up to the end of its last message, the point the next turn extends.
- A conversation's first turn still saves its system head, so new conversations on the same system prompt restore it, unless the head is short beside a long message.
- A new snapshot keeps the previous turn's while the resident budget holds it, so a person who edits their last message resumes there. Eviction drops these kept turns first.
- fix(ggml-cuda): paged attention keeps its partials within 256 MiB. A chunk with many rows covers the context with fewer, longer partitions.
- feat(qwen35): requests decoding beside a long prompt keep half the time, for prompts of more than 8192 new tokens (
StepPlanLimits::time_share_prompt_tokens), so an agent's next step keeps every step. - perf(qwen4exp): an agent's next step no longer gives up time to decoding requests: the same 8192-token size on Flash-Next.
Measured
Each model served with its Manage command on one box (R9700 + Strix Halo, 192 GB), greedy, from a cold start. Scenarios: an 8-turn chat; one agent with tools over this repository (13.7K-token system prompt); four agents at once; a long document and a follow-up while a person chats; a chat on a 4K-token system prompt whose last message is edited and sent again, then a new conversation on the same prompt.
Qwen3.8-Flash-Next (split, 262K context, four slots), main (b8b3cf5c) against this PR:
| Scenario | main | This PR |
|---|---|---|
| Chat: prompt tokens prefilled, 8 turns | 8,465 | 1,951 |
| Chat: time to first token, mean | 1.42 s | 1.26 s |
| One agent: time to first token after step 0, mean / max | 2.92 / 5.07 s | 2.50 / 3.52 s |
| Four agents: wall time | 180.9 s | 159.4 s |
| Four agents: prompt tokens prefilled | 87,082 | 54,918 |
| Four agents: time to first token, mean / max | 3.33 / 7.52 s | 3.29 / 5.95 s |
| Follow-up on a 130K-token document | 102.9 s | 2.7 s |
| Person chatting beside the document: decode | 1.6 tok/s | 19 tok/s |
| Document's first answer, with the person beside it | 111.7 s | 150.9 s |
Qwen 3.8 27B (R9700, 131K context, eight slots, built with Manage's flags for it: HIP graphs off), Manage's pin (e6f643d3) against this PR:
| Scenario | Manage pin | This PR |
|---|---|---|
| 118K-token document sent on its own | server aborts 79 s in (out of memory) | answers in 524 s |
| Follow-up on that document | 532 s | 3.8 s |
| Person chatting beside the document: decode | 0.6 tok/s | 47 to 60 tok/s |
| Document's first answer, with the person beside it | 530 s | 540 s |
| Four agents: wall time | 169.1 s | 121.9 s |
| Four agents: prompt tokens prefilled | 58,792 | 36,457 |
| Four agents: time to first token, mean / max | 16.4 / 43.5 s | 10.8 / 24.0 s |
| Chat, one agent, edit: time to first token, mean | 0.66 / 5.53 / 1.22 s | 0.60 / 4.07 / 1.22 s |
| Published harness, HumanEval prompts (10 x 2, 512 tokens) | 175.3 tok/s | 174.9 tok/s, same text 10/10 |
DeepSeek V4.1 Flash (R9700 + Strix Halo, ds41-gorgon, one request at a time, HIP graphs off as Manage builds it): the follow-up on a 33K-token document restores it and prefills 79 tokens in 1.5 s, against 33,233 tokens in 63 s; its time to first token drops from 72 s to 11 s, the rest being the wait behind a person's request. The agent, edit and chat scenarios take the same time as at Manage's pin (55.6 / 25.3 / 80.4 s against 56.0 / 26.6 / 79.2 s), and the R9700's memory stays at 27.3 to 28.0 GiB as it does there.
Real apps. None of the changes is specific to one app; they apply to any client of the OpenAI or Anthropic API. We checked them with two coding agents against this PR on Flash-Next, Claude Code 2.1.296 on a clean config (22 tools) and Codex 0.155.1: an 11-turn Claude Code task in 73 s (every request failed on main) and a 10-turn Codex task in 84 s. From the second turn on each turn restores the previous turn's end. On Qwen 3.8 27B, the same Claude Code task finishes in 97 s, and Codex in 52 s with 66,895 of its 82,926 input tokens served from the prefix cache.
Costs:
- Snapshot saves. Saving at each turn's end takes about 10 µs per token on Flash-Next (0.29 s at 25K tokens), in that turn's time to first token.
- Memory. Keeping the previous turn holds one more snapshot per conversation, within the resident budget; it is the first to go when the budget is short.
- An edit re-reads the previous reply. The previous turn's snapshot ends before the reply, so an edited last message prefills the reply and the message (223 tokens, 0.78 s on Qwen 27B, against 0.69 s before).
- Output can differ from main. A hit restores at the previous turn's end, so its prefill forwards are cut at different places than a cold run of the same prompt; greedy outputs can differ at near ties.
Tests
| Test | Result |
|---|---|
test_server_unit | 659 passed |
test_parallel_prefix_txn | passes |
New and updated tests: the capture is the end of the last message on every backend; a first turn saves its head; captures in flight hold their slots and bytes; the previous turn survives pruning while the budget holds it and an edit restores it; kept turns are evicted first; the scheduler captures for two requests admitted together, keeps the previous turn and frees the one before; decode-only steps run between the chunks of a long prompt and not beside a prompt under the engine's long-prompt size; a short prompt runs alone before a long one; a strict template gets a late system message as a user turn and a permissive one keeps it; adaptive thinking is the default; the cached-token usage fields.
Built for gfx1151 + gfx1201.
🤖 Generated with Claude Code