feat: TQ3_0 (TurboQuant 3.5bpv) KV cache — 22% memory savings

Merged
DU
6mo ago
+58-26

Summary

Adds TQ3_0 KV cache quantization support, reducing VRAM usage by 22% vs Q4_0 (3.5 vs 4.5 bpv) with identical decode speed. Based on the TurboQuant 3-bit configuration.

This enables 262K context on a single RTX 3090 with 1024 MiB to spare.

What's in this PR

llama.cpp submodule (+1384 lines across 39 files):

  • GGML_TYPE_TQ3_0 type registration, block_tq3_0 struct
  • CUDA quantize/dequantize kernels (cpy, set_rows, getrows)
  • Flash attention vec kernel integration for TQ3_0 K/V
  • Chunked FA kernel for TQ3_0 prefill
  • FWHT (Fast Walsh-Hadamard Transform) rotation kernels
  • 16 FA template instantiations for TQ3_0 type combos
  • CLI: -ctk tq3_0 -ctv tq3_0
  • → See Luce-Org/llama.cpp PR #1

dflash integration (+45 lines across 5 files):

  • DFLASH27B_KV_TQ3=1 env override / --kv-tq3 CLI flag
  • FWHT rotation: Q rotated before FA, output un-rotated from V space
  • 256-aligned cache allocation for TQ3_0 FA stride requirements
  • kv_k_type stored in TargetCache for downstream graph building

Benchmarks — RTX 3090, Qwen3.5-27B

MetricTQ3_0Q4_0Delta
KV @ 64K896 MiB1152 MiB-22%
KV @ 128K1792 MiB2304 MiB-22%
KV @ 262K3584 MiB4608 MiB-22%
Decode~40 tok/s~40 tok/ssame
Prefill~12% slowerbaselineFWHT overhead

Usage

# llama-server (mainline)
llama-server -m model.gguf -c 262144 -ctk tq3_0 -ctv tq3_0 -fa on

# dflash
DFLASH27B_KV_TQ3=1 build/test_dflash model.gguf draft.safetensors prompt.bin 32 out.bin
python dflash/scripts/run.py --model model.gguf --kv-tq3 ...

Test plan

  • llama-server starts and serves inference at 64K, 128K, 262K context
  • dflash test_dflash works with DFLASH27B_KV_TQ3=1
  • Compute-sanitizer (memcheck) passes with 0 errors
  • Decode speed matches Q4_0 baseline