fix(qwen35): trim legacy device pools between requests

Merged
BO
2mo ago
+148-15

Summary

  • add an explicit trim hook for legacy CUDA/HIP device pools
  • release per-evaluation Q8 memo allocations before trimming cached blocks
  • trim target and distinct draft backends when Qwen3.5 request scratch is released
  • extend the existing CUDA pool-shutdown regression test through explicit trimming

Root cause

On non-VMM ROCm devices, long prefills returned operator temporaries to the legacy ggml pool, but the pool retained those blocks indefinitely. A later request with a different allocation shape could require another large contiguous allocation and fatally OOM even though the previous request had released its scratch tensors.

Validation

  • HIP gfx1201 release build completed for dflash_server and test_server_unit
  • two independent back-to-back requests completed on an AMD Radeon AI PRO R9700: 40,013 and 39,234 prompt tokens
  • request cleanup returned 2,405.3 MiB and 2,243.6 MiB from device pools respectively
  • unit suite: 421/423 passed in the restricted workspace; the two socket-permission failures passed when rerun with normal socket permissions
  • git diff --check clean