fix(qwen35): trim legacy pools during long prefills
Summary
- add an opt-in interval for trimming cached allocations from legacy CUDA/HIP pools between completed Qwen3.5 prefill chunks
- reuse the synchronized trim hook introduced by #668, which retires captured graphs and the Q8 memo before returning cached blocks
- document
DFLASH_PREFILL_POOL_TRIM_TOKENS
Problem
#668 trims legacy pools when request scratch is released. A single long, shape-changing prefill can still exhaust a non-VMM HIP device before reaching that request boundary. On gfx1201, a 60,363-token prefill OOMed in the stock runtime even though many retained pool blocks were no longer live.
This change trims only after a chunk has fully completed: target compute and argmax readback are done, the cache position is committed, and draft-side features have been synchronized. It is disabled unless the environment variable is set. VMM backends remain unaffected because ggml_backend_cuda_has_legacy_pool() returns false.
Validation
Hardware and runtime:
- Radeon AI PRO R9700 (
gfx1201, 31.859 GiB VRAM) - ROCm 7.2.4
- Qwen3.8-27B Q4_K_M target with DFlash2 block 16
- Q8 K/V,
--max-ctx 262144, concurrency 1 DFLASH_PREFILL_POOL_TRIM_TOKENS=8192- fresh server for every measurement
| Workload | Stock | Patched |
|---|---|---|
| 60,363 input tokens | OOM | PASS, 125.6 s |
| 60,363 peak VRAM | OOM near card limit | 29.789 GiB |
| 32,781 input prefill | 54.0 s | 53.9 s |
| 32,781 peak VRAM | 30.961 GiB | 29.637 GiB |
| 32,781 output | baseline | byte-identical content SHA-256 |
Long-context patched probes:
| Input tokens | Prefill | Average prefill | Peak VRAM | Free at peak |
|---|---|---|---|---|
| 60,363 | 125.6 s | 480.6 tok/s | 29.789 GiB | 2.070 GiB |
| 100,013 | 270.9 s | 369.2 tok/s | 29.898 GiB | 1.961 GiB |
| 131,013 | 420.3 s | 311.7 tok/s | 30.319 GiB | 1.541 GiB |
| 200,013 | 861.7 s | 232.1 tok/s | 30.273 GiB | 1.586 GiB |
| 258,013 | 1351.1 s | 191.0 tok/s | 30.366 GiB | 1.494 GiB |
During the 258K run, process GTT remained about 8 MiB, swap was disabled, and pswpin/pswpout remained zero. The capacity result therefore did not rely on VRAM-to-RAM paging.
The 60K run logged seven trims at 8,192-token intervals, releasing 238-760 MiB per trim. The direct- and meta-backend allocator shutdown/trim tests passed 2/2 on gfx1201. The patched dflash_server also builds against current main (d227a505) with only pre-existing warnings, and git diff --check passes.
Scope and tradeoffs
This was validated on the single-sequence Qwen3.5 path using a gfx1201 HIP legacy pool. The pool API's meta-backend traversal was exercised through a one-rank meta backend; CUDA, VMM, gfx1100, multi-rank tensor parallelism, and concurrent-serving paths were not runtime-tested. The feature is opt-in because each trim synchronizes the target backend and requires graph recapture; the 32K A/B showed no measurable prefill regression, but very short intervals are not recommended.
The configured 262,144-token window was nearly reached: 258,013 input tokens plus one generated token completed successfully. This does not claim a full 4K-token completion at that prompt length.