feat(qwen35): prefill recall with the exact scorer on the serial path, recall profile on by default
Builds on #934 (merged). Base: main d972beba3. Review range: d972beba3..4f2de18da, 31 commits, 32 files, +2668 / -60. The merge commit 2a01709b0 brings in main 1d949fd88. Its only conflict was test_server_unit.cpp, where #1031 pruned the model-card /props tests next to the three /props.kvflash tests. It takes main's version of that file and keeps the three tests, and it adds no code of its own.
This head replaces the earlier head a21ae4883 (51 commits, +3955 / -190) with three changes:
- Mode 5 is out. Gone: the PFlash drafter as recall scorer, its rescore cut (
LUCE_KVFLASH_M5_RESCORE_TOKENS), the device victim tier (LUCE_KVFLASH_VICTIM_BLOCKS), and the drafter-side PFlash changes only mode 5 needed (1024-token session resume, the device strict-scorer reduce,PFLASH_TW_HOST_REDUCE). Mode 5 failed this PR's gate (P2 19/40 < 22), and on the 2026-11-10 needle replay it kept the needle resident in 0/10 agent sessions against 10/10 for mode 4. PFlash without--kvflashis no longer touched by this PR. - #999 is folded in (last commit):
/props.kvflash {enabled, pool_tokens, mode}, see "/props.kvflash". - The commits were replayed, so the SHAs changed. The map is at the end.
Mode 4, the only recall mode and the one every result below measures, behaves the same as before. On 3 recorded sessions (165 requests, lucebox6 R9700), the earlier head and this head give the same text, tool calls, cached tokens and recall picks on every request, at the same session prefill (640 vs 640 tok/s) and decode (30.1 vs 30.1 tok/s); see "This head against the earlier head". The 14-session gate and the other engine results below ran on the earlier head.
The landing gate passed on the earlier head against both baselines, on 14 recorded coding sessions: mode 4 (the --kvflash default) against K1 (recency pool with this PR's preamble pin) and against K0 (KVFlash 1.0 as shipped without a drafter: recency pool, no pin). P1 31/40 against 0/40 (K1) and 1/40 (K0); P2 26/40 against 24/40 and 19/40; session prefill 0.860x and 0.859x (bar 0.85x). Before the mode-4 cost fix it failed this gate (prefill 0.504x, P2 one probe short).
Why
With --kvflash N on main and no Qwen3.5-0.8B drafter next to the model, the KV cache is a recency pool (policy=lru) of 64-token chunks that keeps only the newest chunks. With a drafter present main picks chunks by drafter scores. Every K arm below ran with no drafter (server log: policy=lru (recency-only: no Qwen3.5-0.8B drafter found ...)). A coding agent that asks about a file it read 60K tokens ago gets a pool without that file. On 14 recorded coding sessions, the far-back recall probe (P1) scored 0/40 on a recency pool with the preamble pin (K1, below).
This PR makes the single-request path bring back the old chunks the new question needs, before it writes the question into the pool. It is the first caller of the #934 scorers. Nothing concurrent is in it: no arena and no slot pagers.
The decode-mask commit 3047a9ff2 is required by recall: a rewind leaves rows above the decode position in the pool.
What it does
Turn by turn, on a coding session with a 16K pool (--kvflash 16384, mode 4):
- Turns 1 to 20: the agent reads files and runs tests. The prompt grows to 60K tokens. The pool holds the pinned system+tools preamble (up to 1152 rows), the sink, and the newest chunks. Old file reads are evicted to host memory.
- Turn 21 arrives: "why does
parse_configignore the second section?" The prompt is the old 60K plus about 400 new tokens. - The prefill is cut at the start of that last user turn. Everything before the cut is written as before.
- First pass: the last user turn runs on its own. The query of its final token is kept, with the last 64 queries of the prompt.
- Recall: those queries, plus the latest question's queries from the store, are scored against every sealed chunk with the exact attention mass (GPU scorer, AVX2 fallback). The best 30 blocks of 2 chunks (60 chunks, the budget 10 x
LUCE_KVFLASH_RECALL_BLOCKS) are picked, ties to the newer block. The pinned preamble is never a candidate. - The state is restored to the cut, the pager is truncated there and reselected: preamble, sink, the picked blocks, then the newest chunks. The picked chunks are paged in from host.
- The last user turn is written again under the new pool, and decoding starts.
Steps 4 to 6 cost one extra forward over the last turn plus one scorer call. The mode-4 cost fix (below) skips them for tool-loop turns under an unchanged question.
Files:
kvflash_prefill_reselect.h: the mode gate (0 and 4; the retired modes 1, 2, 3 and 5 are off), the turn cut,kvflash_recall_pick,kvflash_recall_plan(the budget), the preamble pin helpers, andkvflash_recall_gate. The gate runs a request as a plain recency pool when no sealed chunk precedes the turn, when the chunks up to the cut plus the turn's reserve already fit the pool, or when the pool cannot hold pins + recall + reserve.kvflash_m4_reuse.h(last commit): the decision for the single-pass turn, as pure functions.- Mode 4 (exact scorer, rewind).
--kvflashinstalls it; the startup line saysrecall profile: mode 4 (exact scorer), or names aLUCE_KVFLASH_PREFILL_RESELECTthe environment set. - User-question queries (UQ): the latest human turn's last 64 query rows are kept in a per-session LRU and appended to the scorer's queries (
LUCE_KVFLASH_UQ_QUERIES). - Preamble pin (
LUCE_KVFLASH_PREAMBLE_ROWS, 1152 rows = 18 chunks): the system+tools chunks are pinned, and re-pinned after reset, restore and truncate. The reserve comes out of--kvflash N, so the pool and the decode cost stay at N and the history budget is N - 1152. - Pager:
reselect_recent(); strided page copies and block zeroing (one 2D call per layer and K/V); a chunk recalled from host and not written since skips its D2H copy; page-outs leave the freed block unzeroed; one fence per reselect; victim FIFO compaction; pooled ubatch clamped to the unpinned capacity. - Graph:
TargetCache::q_cap_allholds the lastkQueryCaptureTokens = 64queries of a batch. It exists only in a process with aLUCE_KVFLASHpool, and a graph writes it only on prefill windows that recall mode 4 or UQ read (kvflash_capture_q_all); decode and restore steps never write it. The reader fetches only the captured rows. After a mode-4 rewind the query tail is trimmed at the rewind point and re-captured by the second pass. - Tensor parallelism:
kvflash_recall_features(tp_degree, ask)turns recall and UQ off at TP degree > 1, with the line[kvflash] recall disabled: tensor parallelism has no split for the per-token query capture. Non-qwen35 targets get no recall profile fromkvflash_recall_env. No tensor-parallel run was made. - Decode mask:
kvflash_upload_mask(cur_pos)bounds the slot mask tocur_pos + 1and rebuilds it every decode step (commita261eb59e). It hides rows above the decode position (rejected drafts, an earlier turn's tokens after a rewind) and changes AR numerics for KVFlash decode. It applies only when a KVFlash pool is active.
With --kvflash off the KVFlash paths are not entered: no q_cap_all, no recall branch, no profile env. PFlash is unchanged.
Mode-4 cost fix (160a786b2)
Cause, measured on the 882-request replay of the 14 sessions on the earlier build 787e9f89b (lucebox4, R9700): 810 requests reselected. Mode 4 paid a scorer call (512 s total, mean 632 ms, of which 241 s host-to-device copies) and a rewind that re-prefilled 750,779 tokens (51% of all 1,470,012 new tokens) on each. Prefill was 3609 s against K1's 1821 s, a gap of 1788 s; 1776 s of that gap was on the 810 reselect requests. Skipping the rewind when the picks equal the resident set saves nothing: across 700 consecutive reselect pairs under one question, 0 had identical picks (mean overlap 0.55).
A request now pays the scorer and the rewind only when its latest human question is new. Tool-loop turns under the same question are the bulk of the 810.
- Reuse: every recall candidate (chunk left of the tail window) has an exact score from the last scoring and no scored chunk was rewritten. The turn picks again from those scores. No scorer, no rewind, no recency reset.
- Prescore: a candidate has no score. The exact scorer runs before the pass with the question's stored queries plus the tail queries the previous request left, then picks. No rewind.
- A new question, another conversation, or a rewritten scored chunk takes the unchanged two-pass path.
LUCE_KVFLASH_M4_SINGLE=0forces the two-pass path on every turn. - No candidate is ranked by a score the scorer did not compute for it: an unscored candidate forces a scoring.
On same-question turns the picks can differ from the two-pass picks, because the tail queries can be up to one scoring old (at most one tail window). On the gate replay this cost no recall: P1 31/40 and P2 26/40 with 637 of 882 requests on reuse, against 26/40 and 21/40 for the two-pass mode 4 on the earlier head (other binary, see below).
Measured on the gate (below): 637 requests reuse, 80 prescore, 93 take the two-pass path, 72 skip (pool fits or too small); scorer total 106 s (was 512 s), re-prefill 63,332 tokens (was 750,779); session prefill 0.860x of K1. The simulator estimate before the run was 640 / up to 82 / 88 and 0.86x to 0.90x.
Tests
test_kvflash_m4_reuse has 11 cases, including an_unscored_chunk_left_of_the_window_never_reuses and reuse_pick_ignores_whatever_the_window_chunks_would_score. The test file and the code are in the same commit; the test failed to build before the header existed (kvflash_m4_reuse.h: No such file or directory). Other test commits come before the code they cover; those that failed against their parent say so in the subject (red: ...).
All runs below: RTX 3090 (sm_86, WSL), head 50020e4c5, ctest -j1 --timeout 120. Baseline: main dd85248d3 on the same 3090 (1515 tests, 7 failed; cuda3090/main-ctest.log). Main d972beba3 was not run as a full ctest.
| Check | Result |
|---|---|
test_server_unit KvflashM4Reuse | 11 passed, 0 failed |
test_server_unit Kvflash | 280 passed, 0 failed |
full ctest | 7 failed of 1707. 6 of the 7 also fail on the main baseline: SampledVerifyFixture.constrained_tokens_match_constrained_ar_sampling, SampledVerifyFixture.committed_tokens_match_ar_sampling and SampledVerifyFixture.redrawing_a_drawn_row_is_caught (all three time out at 120 s on main too), test_ds41_mmid_width_invariance, qwen4exp_prefix_cache, server_unit_entrypoint. Only QnSplitFusionsFixture.q4k_rows (timeout at 120 s, box shared) is outside the baseline; its rerun alone on this head and on d972beba3 is pending. None is a KVFlash test. |
full ctest with CUDA_VISIBLE_DEVICES="" | 18 failed of 1707 (GPU tests without a GPU). No CPU-only baseline exists. |
CI on the earlier head 2f0b18313: Linux build, Windows MSVC + CUDA, Windows ROCm HIP; GPU tests on DGX GB10 (sm_121), R9700 (gfx1201), Strix Halo (gfx1151) | pass. On this head a21ae4883 (run 43282854502): Linux build, Windows MSVC + CUDA, Windows ROCm HIP, DGX GB10 and Strix Halo pass. The R9700 job failed: the runner got a shutdown signal mid-build (no test ran) and needs a re-run. |
Engine run
Arms. K1 = LUCE_KVFLASH_POLICY=lru with this PR's preamble pin on (its server log shows preamble pin end=). K1 is not KVFlash 1.0 as shipped. K0 = LUCE_KVFLASH_POLICY=lru LUCE_KVFLASH_PREAMBLE_ROWS=0 (pin off, the 1.0 pool), run on this head (below). M4 = mode 4 (--kvflash default).
This head against the earlier head (identity)
lucebox6 R9700 (gfx1201), 2026-11-18, Qwen3.8-27B UD-IQ4_XS (/opt/models, the gate's model). HIP builds (gfx1151;gfx1201) of the earlier head a21ae4883 (luce_server md5 78bfef46) and this head 4f2de18da (md5 269cdd9a). Flags on every arm, no env: --kvflash 16384 --max-ctx 139264 --cache-type-k q8_0 --cache-type-v q8_0 --prefix-cache-slots 8 --prefix-cache-max-mib 40960 --disk-prefix-cache off --no-kv-cache --target-device hip:0. 3 of the 14 recorded sessions (165 requests, P1 and P2 probes included), temperature 0, seed 1, thinking off. Arms ran in the order old, lean, lean, old, one server at a time, page cache dropped before each.
| Arm | Same output as old1 | Session prefill | Decode med / mean / best / min (n 10, decode > 2 s) | Wall |
|---|---|---|---|---|
old1 (a21ae4883) | - | 641 tok/s | 30.05 / 30.04 / 30.20 / 29.90 | 632.8 s |
| lean1 (this head) | 165/165 | 640 tok/s | 30.10 / 30.06 / 30.20 / 29.90 | 633.7 s |
| lean2 (this head) | 165/165 | 639 tok/s | 30.10 / 30.05 / 30.20 / 29.80 | 634.1 s |
old2 (a21ae4883) | 165/165 | 640 tok/s | 30.10 / 30.06 / 30.20 / 29.90 | 633.6 s |
Same output = same text (sha256) and the same tool-call names and arguments; prompt and cached tokens are also equal on 165/165. The 657 [kvflash] selection lines (153 recall picks of 30 blocks) are identical across all four arms once timings are masked and the earlier head's (0 tier hits) suffix is dropped: the device victim tier never hit in mode 4 (186 lines, all 0). Every arm logs recall profile: mode 4. old1 shared the GPUs with a unit-test run for about 30 s near its start. /props on this head: "kvflash": {"enabled": true, "pool_tokens": 16384, "mode": "prefill-reselect=4"}.
HIP unit tests, same builds, ctest -R "[Kk]vflash|[Ll]aunch|[Pp]rops|[Ff]eature.?[Gg]ate": this head 284/284 on gfx1201 (R9700) and 284/284 on gfx1151 (Strix Halo); the earlier head 299/299 on both (it still has the mode-5, victim-tier and probe tests, and not the 3 /props tests). Every commit of this head builds (CPU, CUDA toolchain).
Landing gate on the earlier head (failing)
lucebox4 R9700 (PCIe 3.0 x2), 2026-11-10, Qwen3.8-27B UD-IQ4_XS (md5 54db94b1), 14 recorded sessions, 40 P1 and 40 P2 probes, 882 requests per arm. Binary luce_server md5 a2f83fbd built from 787e9f89b (pre-rebase). It differs from 2f0b18313 only in the --kvflash default-mode selection (launch_profiles.h, server_main.cpp); the K1 and M4 arms have no drafter and are unaffected.
Parity: --kvflash 16384, --max-ctx 139264, q8_0 K and V, prefix cache on (40960 MiB, 8 slots, disk off), thinking on, T 1.0, top_p 0.95, top_k 20, effort medium, probe max_tokens 16384, same sessions on every arm. Caching state: M4's prefix_len histogram differs from K1's on 50 of 882 requests (total 4,106,452 against 4,106,602 tokens). No power limit was set or recorded. Session prefill = total new tokens / total prefill seconds. Wall = sum of client turn seconds.
| Arm | P1 | P2 | Session prefill | Decode median (out >= 64) | Wall |
|---|---|---|---|---|---|
| K1 (lru + pin) | 0/40 | 24/40 | 807 tok/s | 29.0 tok/s | 3312 s |
| Mode 4 | 26/40 | 21/40 | 407 tok/s (0.504x) | 28.8 tok/s | 5791 s |
Mode 4: P1 passes (>= K1 + 3), P2 fails (21 < 22), prefill fails (0.504x < 0.85x). This table is the cost the fix above targets; it is not a result for this head. P1 is 0/40 on K1 and 26/40 on mode 4 on the same box and binary. K0 did not run on that head; it ran on this one (below).
Re-gate on this head (passing)
lucebox4 R9700 (PCIe 3.0 x2), 2026-11-11, head 50020e4c5, one luce_server (md5 ecbe4373) for both arms, Qwen3.8-27B UD-IQ4_XS (md5 54db94b1), the same 14 sessions (882 requests, 40 P1 and 40 P2 probes per arm). Server flags on both arms: --max-ctx 139264 --prefix-cache-max-mib 40960 --prefix-cache-slots 8 --disk-prefix-cache off --no-kv-cache --cache-type-k q8_0 --cache-type-v q8_0 --kvflash 16384. Runner: thinking on, effort medium, T 1.0, top_p 0.95, top_k 20, probe max_tokens 16384. Arms differ only in env: K1 LUCE_KVFLASH_POLICY=lru; K0 LUCE_KVFLASH_POLICY=lru LUCE_KVFLASH_PREAMBLE_ROWS=0; M4 none. The K1 and M4 server logs show the preamble pin (851 lines each), K0's none. Caching state: prompt tokens 62,012,284 on every arm, cached prefix 60,542,275 (K0, K1) vs 60,542,272 (M4), equal on 881 of 882 requests against each. No power limit set or recorded. Gates were fixed before any run (m4cost_gate.sh).
| Arm | P1 | P2 | Session prefill (all) | Per-session prefill med / mean / best / min | Decode, out >= 64: med / mean / best / min | Wall |
|---|---|---|---|---|---|---|
| K0 (1.0: lru, no pin) | 1/40 | 19/40 | 811 tok/s (1812 s) | 818 / 813 / 879 / 715 | 29.1 / 29.2 / 29.5 / 29.0 (n 79) | 4241 s |
| K1 (lru + pin) | 0/40 | 24/40 | 811 tok/s (1813 s) | 818 / 813 / 876 / 715 | 29.2 / 29.2 / 29.6 / 29.0 (n 80) | 3347 s |
| Mode 4 | 31/40 | 26/40 | 697 tok/s (2109 s), 0.860x | 718 / 704 / 790 / 589 | 29.0 / 29.0 / 29.4 / 28.7 (n 71) | 3718 s (1.11x) |
Gate lines, M4 vs K1: prefill >= 0.85x PASS (0.860); P1 >= K1 + 3 PASS (31 vs 0); P2 >= K1 - 2 PASS (26 vs 24). M4 vs K0: prefill >= 0.85x PASS (0.859); P1 >= K0 + 3 PASS (31 vs 1); P2 >= K0 - 2 PASS (26 vs 19). Wall is not engine speed here: it includes the probes' answers, and K0's probes generated 63,116 output tokens against 37,166 (K1) and 39,113 (M4). Mode-4 paths: reused 637, prescore 80, full 93, skip-fits 21, skip-pool 20, none 31; scorer 106.3 s, paging 28.0 s, re-prefill 63,332 tokens.
Screen before the gate (3 of the sessions, 151 requests, same box): this head vs its parent a90f8f1de (md5 33daf153): session prefill 737 vs 443 tok/s (0.883x vs 0.530x of K1's 835), scorer 23.5 s vs 96.0 s, P1 9/9 on both. With LUCE_KVFLASH_M4_SINGLE=0 the head's picks equal the parent's on 25 of 25 requests (the two-pass path is unchanged).
Versus main (same box, serial, speed and text)
lucebox6 R9700 (gfx1201), ROCm 7.2.4, 2026-11-11, Qwen3.8-27B UD-IQ4_XS (md5 54db94b1), q8_0 K/V, --max-ctx 131072 --prefix-cache-slots 0 --no-kv-cache --kvflash 16384, temperature 0, no draft model in either arm. Serial engine; 3 conversations, turn 1 21585-26509 and turn 2 21758-26682 prompt tokens, prefix_len=0 on both sides (full reprocess, no caching). Two server starts per pin, interleaved; n = 6 per pin and turn. main d972beba3 (luce_server md5 f2f9ae3f) against 50020e4c5 (md5 b38275f8; it differs from this head only in ENVIRONMENT.md, a comment and one unit test).
| Pin | Turn | Prefill tok/s med (mean, best, worst) | Decode tok/s med | Wall s med |
|---|---|---|---|---|
| main | 1 | 901.1 (900.4, 903.6, 897.0) | 29.5 | 30.0 |
| this PR | 1 | 880.3 (881.7, 886.2, 878.6), -2.3% | 29.3 | 30.7 |
| main | 2 | 899.2 (897.7, 899.6, 894.1) | 29.5 | 30.3 |
| this PR | 2 | 861.0 (861.0, 861.2, 860.6), -4.2% | 29.3 | 31.5 |
The text differs from main on all 6 requests, and each side repeats its own text on both starts: main logs policy=drafter (attaches on first reselect) and, with no drafter, keeps the newest chunks; this PR logs recall profile: mode 4 / policy=qk and picks chunks by score. This run has no recall probes; recall against KVFlash 1.0 is the K0 row of the gate above.
Versus full KV (no KVFlash)
These compare against no KVFlash, not against KVFlash 1.0, and none was measured on this head. They also ran with the preamble reserve outside --kvflash N on earlier binaries, so the pool size differs from this head (see the LongMemEval-S line).
E20 (20 live coding-agent tasks), lucebox4, 2026-11-10, luce_server md5 6ae4a114 (commit e964d21a7; not the binary of the gate table above, which is a2f83fbd / 787e9f89b; --kvflash there installs mode 4). Server flags: --max-ctx 135168 --prefix-cache-max-mib 12288 --prefix-cache-slots 4 --disk-prefix-cache off --cache-type-k q8_0 --cache-type-v q8_0 --no-kv-cache, --kvflash 16384 on the K arms. Runner: thinking on, effort medium, T 1.0, top_p 0.95, top_k 20, seed 1, max_tokens 16384, same 20 tasks on every arm.
| Arm | Pass | Sum client wall | Session prefill | Median decode |
|---|---|---|---|---|
| Full KV | 20/20 | 7002 s | 494 tok/s | 19.1 tok/s |
| Mode 4 | 19/20 (p02) | 4572 s | 690 tok/s | 28.9 tok/s |
| K1L (lru + pin) | 19/20 (p02) | 4261 s | 859 tok/s | 29.1 tok/s |
Full KV is at ceiling on this set (it fails the pre-registered 25-85% band), so the set measures no regression against full KV and speed, not recall. It cannot show a recall edge over recency: mode 4 and K1L tie. Mode 4 session prefill is 0.80x of K1L and wall +7%.
EXECODE A6 (2026-10-27, lucebox6 R9700, binary md5 0ad28447, --max-ctx 135168, q8_0 K/V, 20 paired tasks, pre-registered): 19/20 against 20/20 for full KV, difference -0.05, bootstrap CI [-0.15, 0.0], exact McNemar p = 1.000, parity PASS (frozen margin d <= 2). Medians: decode 19.2 to 27.6 tok/s, request wall 23.0 to 12.9 s, total wall 7,062.5 to 5,135.6 s, first-request prefill (>50K new tokens) 566.2 to 749.6 tok/s.
LongMemEval-S (binary md5 cec754b6, thinking off, T 0, seed 1, max_tokens 512, judge gpt-4o-2024-08-06, n = 500, one request with the whole history): --kvflash 16384 against full context is a measured loss, -8.0 points [-11.4, -4.6] with official labels, -8.8 [-12.0, -5.8] with audited labels (pre-registered verdict: accuracy loss). That binary reserved 6144 extra preamble rows on top of the pool; this head keeps 1152 inside N. Neither A6 nor LongMemEval-S bears on this head's pool size.
Long cold turn (724119d03, was a21ae4883)
With the --kvflash profile, a single cold turn longer than the pool skipped the recall: its user turn starts in chunk 0, so nothing is sealed before R, and the turn is longer than the recall gate's PoolTooSmall bound. The turn then ran LRU only, and a fact the pool dropped from the middle of the prompt was gone at decode. kvflash_max_turn_tokens is that bound. kvflash_mode4_plan takes it as an optional cap and rewinds a longer turn to its tail, so the recall scores and pages in everything before it. The gate is unchanged and a turn within the cap keeps its R. A capped R can land inside the latest question's 64-query window, where a replay would recompute the window's tail while the stored question rows keep the old pool's values; kvflash_uq_snap_rewind moves such an R to the window end.
Serial needle on the RTX 3090: Qwen3.8-27B IQ4_XS, DFlash2 drafter, --max-ctx 131072, --kvflash 16384, q8_0 K and V, temperature 0, the shipped --kvflash profile with nothing else set, one cold 81,430-token turn of 3,500 records ("record i: the code for key_i is ..."), question at the end, cached tokens 0, one run per head.
| head | server log | key_123 (want 74010) | prefill |
|---|---|---|---|
d98512ff0 | prefill reselect mode=4: skip, no sealed chunk before R (LRU) | 10098, wrong | 74.4 s |
a21ae4883 | prefill reselect mode=4: R4=70230 (end 81430, re-prefill 11200 tokens) | 74010, right | 87.4 s |
The 13 s difference is the scoring and the 11,200-token replay. One key, one run per head.
Second shape, the three-turn serial smoke (--kvflash 16384, same model and flags; turn 1 is a cold 50,408-token request with two user messages, the last one longer than the cap; one run per head):
| head | turn 1 (want 78146) | turn 2 (want 17216) | turn 3 (want 75631) | turn 1 prefill |
|---|---|---|---|---|
d98512ff0 | 74084, wrong | right | right | 48.3 s |
43ebed574 | 74084, wrong | right | right | 51.6 s |
a21ae4883 | 78146, right (R4=39208, re-prefill 11200) | right | right | 56.8 s |
What ran on the commits after d98512ff0 (RTX 3090):
test_server_uniton CPU: 1382 passed, 0 failed, 17 skipped ona21ae4883(1374/0/17 ond98512ff0).test_server_uniton the GPU: 1390/0/6 on43ebed574(1385/0/6 ond98512ff0).- The three-turn smoke above: the same answers and the same 30 picks on
43ebed574andd98512ff0.
Not run on them: ctest, the landing gate, ROCm, tensor parallelism.
/props.kvflash (was #999)
/props gains kvflash {enabled, pool_tokens, mode}, so a client can see whether recall is on without reading the server log. server_main fills it after the --kvflash* flags have set the environment the backend reads, so env-only LUCE_KVFLASH counts too. pool_tokens is LUCE_KVFLASH (0 = auto, sized at load). mode comes from the backend's own kvflash_reselect_mode over LUCE_KVFLASH_PREFILL_RESELECT, LUCE_KVFLASH_POLICY and LUCE_KVFLASH_RESELECT_SCORER: prefill-reselect=4 while recall runs, empty on another architecture, with --kvflash-policy lru or with a scorer other than exact. It does not see tensor parallelism, which turns recall off later at load. #999's nested recall {protocol, active, mode} object and the --help marker are not carried: no client reads them.
Env knobs
--kvflash on a qwen35 target installs mode 4: LUCE_KVFLASH_POLICY=qk, RECALL_BLOCKS=6, SCORE_POOL_RECENT=1, SELECT_UNIT=2, PREAMBLE_ROWS=1152, PREFILL_RESELECT=4, RESELECT_SCORER=exact, UQ_QUERIES=64. A variable that is already set keeps its value. The values are the named constants of KvflashRecallProfile in launch_profiles.h. Other architectures install nothing. test_launch_policy.cpp pins the full (name, value) set of both profiles.
New knobs documented in server/docs/ENVIRONMENT.md: the recall knobs listed there. LUCE_KVFLASH_M4_SINGLE (0 forces the two-pass path) is read in qwen35_backend.cpp and is not yet in ENVIRONMENT.md.
Limits
Not run on this head (they ran on the earlier head, whose mode 4 this head matches request for request on the 3-session replay):
- The 14-session landing gate.
- E20, EXECODE and LongMemEval-S on this head.
- Any engine run of the cost fix on CUDA. The RTX 3090 results are unit tests and ctest only; the engine numbers are R9700 (HIP).
- HIP unit tests and Strix (gfx1151) / DGX GB10 results (Strix and DGX GB10 pass on this head's CI; the R9700 job is to be re-run; no local run).
- A main-vs-branch run with recall probes; the serial run against main measures speed and text only.
- Tensor parallelism.
- The ctest baseline is main
dd85248d3, notd972beba3; the alone rerun ofq4k_rowsis pending (one alone attempt ofcommitted_tokens_match_ar_samplingis logged with rc=77 and no pass or fail count, so it is not counted). No CPU-only ctest baseline. - A power limit is not recorded for any engine run.
Commit map (earlier head -> this head)
| Earlier | This head | Subject |
|---|---|---|
a261eb59e | 3047a9ff2 | the KVFlash decode mask hides rows above the decode position |
50020e4c5 | 160a786b2 | mode 4 runs a turn once while the human question is unchanged |
2a6e64c53 | b6d1bea00 | LUCE_KVFLASH_RECALL_BLOCKS is a 10 x N chunk budget |
9fc7a36a4 | 3a8a77357 | the pinned-preamble recall fixture carves the reserve inside the pool |
d98512ff0 | 05c4c3a31 | the per-process q_cap_all reservation under tensor parallelism |
d13a91857 | 99cede4b6 | recapture the turn's tail queries after a rewind |
89a1c9ffa | dbde568ea | write q_cap_all only when recall or UQ is on |
68e5485a1 | 7d435c8e7 | kvflash help default and the resolved reselect mode in the startup line |
43ebed574 | fba9ef4a2 | read only the captured rows of q_cap_all |
a21ae4883 | 724119d03 | recall a turn's tail when the whole turn cannot be replayed |
| (new) | efec5616a | remove mode 5 |
| (#999) | 4f2de18da | /props.kvflash reports whether recall is on |