feat(kvflash): exact attention-mass chunk scorers on CPU and GPU, user-question query helpers, mode 5 drafter scorer

Merged
DU
9d ago
+2486-13

Why

A KVFlash recall step has to rank evicted chunks by how much attention a question would give them. Main has no scorer for that quantity: its selectors rank chunks by recency or by a pooled-key proxy or by the residency drafter, and none of them computes the softmax mass a chunk receives from the question's queries. A caller that wants recall before write (the PR that follows) has nothing to call.

This PR adds the scorers and the helpers around them. It has no caller: with KVFlash off, with KVFlash on, and with any recall setting, no code path in this PR runs, so there is nothing to reproduce on main. The tests below are red on main because the headers they include do not exist there.

What it does

All paths are under server/src/ unless a test is named.

  • common/kvflash_exact_mass.h. The quantity and the types. For one query, the softmax mass each sealed chunk receives over all sealed chunks, averaged over the full-attention layers and query heads: score[c] = mean over (layer, head) of exp(lse_c - Z), with lse_c the log-sum-exp of the chunk's scores and Z the log-sum-exp over chunks. KvFlashMassPlanes also carries the mean of the 8 largest plane masses and the largest one. The plain reference scorer lives in test/unit/kvflash_mass_synth.h and is the oracle for the other two.
  • common/kvflash_exact_fast.h. The AVX2 CPU scorer, one or many queries, on any K type ggml has a to_float for (f32, f16, q8_0, q4_0 and so on). It uses common/parallel_for.h.
  • common/kvflash_gpu_mass.h. The same outputs on the target backend, built from stock ggml ops only (no ggml-cuda.h), so it runs on CUDA, HIP, Vulkan and, in the unit test, the ggml CPU backend. Resident chunks are read in place from the pool tensors; evicted chunks are gathered from their host copies into one pinned staging buffer and uploaded once per call. Queries are scored in balanced batches under a byte budget. Device memory is call-scoped. The call refuses (returns false) when the working set exceeds the device's free memory minus a margin, because WSL spills over-committed VRAM into system RAM without an error.
  • common/kvflash_uq.h, common/kvflash_query_capture.h. Pure helpers for using the latest human question as extra selector queries: the human-turn span (the last user turn that is not a tool result), the capture plan, the per-session store with its key, and the gate that needs every chat token id. kQueryCaptureTokens is 64.
  • common/kvflash_block_scores.h. Per-block sums of chunk scores.
  • common/kvflash_pflash_map.h, pflash/kvflash_drafter_scorer.{h,cpp}. Mode 5: KvFlashPFlash2Scorer runs the PFlash request path as a recall scorer and maps the segment densities to chunk scores by the largest density of the segments that overlap a chunk. It is off unless selected and has no accuracy study. KvFlashDrafterScorer's id folding moves into a helper with the same behaviour.
  • common/kvflash_pool_flag.h. kvflash_pool_requested moves out of kvflash_pager.h so the query-capture header can use it without the pager's ggml include. The function is unchanged. common/fnv.h holds the FNV-1a helpers; it is byte-identical to the file in the pager-slices PR.

Merge order: this branch is cut from main c5179e4b5 and does not need #932. Both touch the include block of kvflash_pager.h and the merge has one conflict there (this PR's #include "kvflash_pool_flag.h" next to #932's #include "fnv.h"); keeping both lines resolves it. fnv.h is added by both with the same content.

Tests

test_server_unit gains 62 cases in four files: KvflashExactMassFixture 20 (match against the reference on f32, f16, q4_0 and other types and GQA shapes, absent chunks, multi-query equals single, exact ties keep the newer chunk, plane aggregates), KvflashGpuMassFixture 12 (mixed residency, all resident, all evicted, fewer than 64 queries, masked free and beyond-sealed blocks, reuse and growth, bad inputs, batch plan and batched equals one batch, parity with the CPU scorer), KvflashUqFixture 14 and KvflashPflashMapFixture 16. Eight of the 62 come from the review round (51423315f): run_workers with n <= 0 and with a throwing worker, GQA dimensions that would drop query heads, a null query on both scorers, max == 0 in the question store, a malformed PFLASH_DRAFTER_SESSIONS, and an empty final message. The two bench cases in KvflashExactMassFixture return at once unless LUCE_EXACT_BENCH is set, so they pass without measuring.

Written red first: each of the four test files was compiled against main c5179e4b5 and stops at fatal error: kvflash_pflash_map.h: No such file or directory (and the same for kvflash_exact_mass.h and kvflash_exact_fast.h), because the headers are new. That is a compile error as the red, since there is no behaviour on main to fail.

test_server_unit, passed / failed / skipped, main c5179e4b5 vs branch 51423315f. There is no separate target; the four files are in the test_server_unit source list. Each of the three commits builds alone on CUDA.

buildmain c5179e4b5branch 51423315f
CUDA, RTX 3090 (sm86, Release)1224 / 0 / 61286 / 0 / 6
HIP gfx1201 (hip:0)1207 / 0 / 41269 / 0 / 4
HIP gfx1151 (hip:1)1207 / 0 / 41269 / 0 / 4

The GPU was idle before each CUDA run and one suite ran at a time. Each HIP log names its device (gfx1201 on hip:0, gfx1151 on hip:1).

Engine run

The engine does not call anything in this PR, so this run shows only that the serial --kvflash path is unchanged. Qwen3.8-27B (Qwen3.8-27B-UD-IQ4_XS.gguf, md5 54db94b16b77e3fe024faa893cc4b80d), luce_server <model> --max-ctx 65536 --kvflash 16384 --no-kv-cache --cache-type-k q8_0 --cache-type-v q8_0, two distinct prompts of 34583 and 37293 tokens, 128 max tokens, temperature 0, seed 1, HIP only. The pool pages out on every arm (pooled prefill: 34583 tokens through a 16384-token pool, and the same for 37293). One server at a time, caches dropped before each arm, arms in the order main, branch, main repeat on each GPU. luce_server md5: main e01ec7db3b155947fc93c00f2acb4a07, branch 15876f7707214e852c43a1483df2a243. Every request shows prefix_len=0 and disk_hit=false on all six arms. The prompts were answered in 61 and 61 tokens on gfx1201 and 61 and 63 on gfx1151 (finish=stop), the same on every arm of a GPU.

With two requests, median equals mean, and best and min are the two requests (for wall, best is the fastest). Speeds are from the server's chat DONE lines; prefill time is printed in steps of 0.1 s.

gfx1201 (hip:0):

armprefill tok/s med / mean / best / mindecode tok/s med / mean / best / minwall s med / mean / best / mintext vs main
main867.78 / 867.78 / 879.55 / 856.0129.35 / 29.35 / 29.40 / 29.3043.65 / 43.65 / 42.67 / 44.63-
branch861.54 / 861.54 / 873.37 / 849.7129.25 / 29.25 / 29.30 / 29.2043.95 / 43.95 / 42.96 / 44.94identical, both requests
main repeat852.32 / 852.32 / 867.28 / 837.3629.15 / 29.15 / 29.20 / 29.1044.42 / 44.42 / 43.57 / 45.27identical, both requests

gfx1151 (hip:1):

armprefill tok/s med / mean / best / mindecode tok/s med / mean / best / minwall s med / mean / best / mintext vs main
main292.85 / 292.85 / 294.57 / 291.1212.90 / 12.90 / 12.90 / 12.90127.67 / 127.67 / 122.21 / 133.13-
branch291.30 / 291.30 / 292.83 / 289.7712.90 / 12.90 / 12.90 / 12.90128.33 / 128.33 / 122.91 / 133.74identical, both requests
main repeat291.90 / 291.90 / 293.57 / 290.2212.90 / 12.90 / 12.90 / 12.90128.06 / 128.06 / 122.65 / 133.47identical, both requests

Branch against the first main run, medians: on gfx1201 prefill -0.72%, decode -0.34%, wall +0.69%; main against its own repeat is -1.78%, -0.68% and +1.76%. On gfx1151 prefill -0.53%, decode 0.0%, wall +0.52%; main against its own repeat is -0.32%, 0.0% and +0.31%. The branch sits inside the spread of main against itself on both GPUs. This PR makes no speed claim.

The tables above are for ca6b23ae9. The review fix 51423315f was run again with the same harness, main (same binary) then branch on each GPU, with no main repeat. Branch luce_server md5 1c904937801c744c586cdbac76c5e228. Text is identical to main on both requests on both GPUs. prefix_len=0 and disk_hit=false on all four arms, and every arm logs pooled prefill for both prompts. Branch vs main, medians: gfx1201 prefill 865.67 vs 867.78 tok/s (-0.24%), decode 29.35 vs 29.35 (0.0%), wall 43.72 vs 43.66 s (+0.14%); gfx1151 prefill 290.89 vs 291.05 (-0.05%), decode 12.90 vs 12.90 (0.0%), wall 128.45 vs 128.41 s (+0.03%).

The server has no logprobs, so near-tie margins between the two builds' token choices cannot be computed; text identity is the comparison.

Limits

  • No caller. Nothing in the server or the backends uses these scorers in this PR, so the engine run says only that the existing path is unchanged. The end-to-end effect is in the PR that calls them.
  • Decode rests on 61 to 63 generated tokens, two requests per arm. It is not a decode benchmark.
  • The engine run is HIP only (gfx1201 and gfx1151), serial --kvflash. There is no CUDA engine run. The unit tests run the GPU scorer on the ggml CPU backend only (ggml_backend_cpu_init), so the device path of kvflash_gpu_mass.h on CUDA and HIP is built here but not run by any test or by the engine.
  • Mode 5 (KvFlashPFlash2Scorer) is off unless selected and has no accuracy study. Its tests cover the density-to-chunk mapping, the message map and the gate, not recall quality.
  • The AVX2 scorer falls back to scalar code when AVX2 is unavailable; the reference scorer is the oracle for both.
  • The merge with #932 has one trivial conflict in kvflash_pager.h (see above).