feat: add TQ3_0 (TurboQuant 3.5bpv) KV cache type
Summary
Adds TQ3_0 quantization support for KV cache, enabling 22% memory reduction vs Q4_0 (3.5 vs 4.5 bits-per-value) with comparable decode speed.
Based on the TurboQuant paper's 3-bit configuration.
Implementation
block_tq3_0struct: 14 bytes per 32 elements, group-quantized with FWHT (Fast Walsh-Hadamard Transform) rotation- CUDA kernels: quantize/dequantize (cpy), set_rows, getrows
- Flash attention: vec kernel integration (
vec_dot_KQ,dequantize_V) for TQ3_0 K/V types - Chunked FA kernel: auto-selected for TQ3_0 prefill (fallback when MMA tensor core kernel is unavailable)
- FWHT rotation: K/V stored in rotated space, Q rotated at attention time, output un-rotated
- CLI flag:
-ctk tq3_0 -ctv tq3_0
Benchmarks — RTX 3090, Qwen3.5-27B
| Metric | TQ3_0 | Q4_0 | Delta |
|---|---|---|---|
| KV @ 64K | 896 MiB | 1152 MiB | -22% |
| KV @ 128K | 1792 MiB | 2304 MiB | -22% |
| KV @ 262K | 3584 MiB | 4608 MiB | -22% |
| Decode | ~40 tok/s | ~40 tok/s | same |
| Prefill | ~12% slower | baseline | FWHT overhead |
Files Changed (39 files, +1384/-16)
Core type: ggml.h, ggml.c, ggml-common.h — type registration, block struct
CUDA quantize: tq3-quant.cuh — centroids, FWHT, quantize/dequantize
CUDA cpy: cpy.cu, cpy-utils.cuh — f32→TQ3_0 copy kernel
CUDA set_rows: set-rows.cu — custom SET_ROWS for TQ3_0 (group-based quantization)
CUDA FA: fattn.cu, fattn-common.cuh, fattn-vec.cuh, fattn-chunked.cu/cuh — flash attention integration
CUDA FWHT: turbo-wht.cu/cuh — FWHT rotation kernels
Template instances: 16 new vec FA template instantiations for TQ3_0 type combos
llama: llama-kv-cache.cpp — skip standard attn_rot for TQ3_0 (has its own FWHT)
CLI: arg.cpp — add tq3_0 to allowed KV cache types
Test plan
- llama-server starts at 64K, 128K, 262K context with
-ctk tq3_0 -ctv tq3_0 - Inference produces coherent output
- Compute-sanitizer (memcheck) passes with 0 errors
- Decode speed matches Q4_0 (~40 tok/s)