feat(kvflash): pager slices, step fences, deferred free and the KVFLASH2 blob
Why
KvFlashPager on main (server/src/common/kvflash_pager.h) is a pager for one pool that starts at row 0 of the KV allocation and frees a block the moment it is paged out. Work that needs more than that has nothing to build on:
- A pool cannot be a slice of a larger KV allocation.
block_of,slot_of,fill_slot_posandfill_slot_maskall assume the pool begins at row 0, so two pools cannot share one allocation. - A page-out frees its block at once (
page_outon main pushes tofree_blocks_directly). A caller that queues page-outs a step ahead and fences once per step has no way to hold the block until the copy has landed. - There is no pin that lasts one step, and no check that a page-out planned for a request is still valid when it runs.
page_out(c)on main does not checkis_pinned(c); only eviction skips pinned chunks (mainkvflash_pager.h:415-416).reselect()sorts by score alone (main:566,a.s > b.s), so on an exact tie across the resident/host boundary the order is whateverstd::sortleaves. A tie can cause a recall and an eviction for the same bytes.- The pooled blob is written only as KVFLASH3, as one flat buffer. A writer that keeps chunks apart cannot produce it without assembling a second full copy.
This PR adds the interface for those and nothing that only a later feature uses. With KVFlash off, no code path here runs. With KVFlash on and the serial path, the decisions are the same except for the three differences listed under Limits.
What it does
Line numbers are in server/src/common/kvflash_pager.h at 25997804c unless a file is named.
- Slices.
KvFlashConfig::row_base(:103) makes the pool a slice of a larger KV allocation. Rows and block ids the backend sees are arena space:block_of(:685),slot_of(:703),fill_slot_pos(:711),fill_slot_mask(:758),arena_tokens()(:695). The serial pagers stay at base 0, where these return what they returned before.attachchecks the slice against the tensors (:132-146) before it touches any state. - Step pins and generation.
pin_step_range/end_step(:459-461) sit beside the existingpin_range(:422).generation()(:558) withpage_out_checked(:555) makes a page-out planned against a request that has since ended do nothing.abort_step(:563). - Deferred free (
:635-638).set_deferred_free,last_record,pending_free_blocksandcomplete. With it on, a paged-out block goes to a pending list (:543) and becomes allocatable only aftercomplete(record)for the record that freed it. It is off by default; the serial path keeps one fence per reused block. - Stream hooks.
set_external_page_stream(:581),page_stream(:573),set_batch_submit(:572). - Selection.
lru_victims(n)(:816).reselect(tail_override)(:775): a tail override of N protects only the last N chunks for that call, and a score tie keeps the resident chunk (:790). A truncate to a prefix that fits the pool callsrelayout_identity_(:284,:297) so the block map is the one a fresh pager would have; pool attention reduces over physical slots in order, so the layout is part of the result. fill_slot_mask(dst, valid_upto)(:758): rows at logical positions at or above the bound are masked off. The default,INT64_MAX, keeps every resident row visible, so existing callers are unchanged. A decode step at position p would pass p + 1.page_outrefuses a pinned chunk (:530).kvflash_require_base_zero(:1633) prints a message and callsstd::abort()when a pager has a nonzerorow_base. It guards the two serial-only consumers inserver/src/qwen35/qwen35_backend.cpp(identity prefill sync:2884, single-mask upload:2965).server/src/common/layer_split_kvflash.h:31makes the identity sync return false on a slice, and the shard daemon's identity-sync condition (server/src/qwen35/qwen35_target_shard_ipc_daemon.cpp:585) addsrow_base() == 0. No path in this PR creates a slice, so none of these fire here. The slot-position buffer inserver/src/qwen35/layer_split_forward.cpp:77is nowarena_tokens()long, which equals the pool size at base 0.server/src/common/fnv.hholds the FNV-1a step used by the pager's ledger hash in its log line (:846), replacing the inline constant main has at:596.
KVFLASH2 and KVFLASH3
serialize() (:893) keeps writing KVFLASH3, the pooled snapshot with the pool layout, byte for byte. snapshot_save_pooled_at and kvflash_restore_pooled are not changed. KVFLASH2 is the same blob without the pool layout, produced in parts by serialize_to (:946), blob_header (:961), read_chunk (:972) and ledger_blob (:996), for a writer that keeps chunks apart. deserialize (:1025) reads both by magic: KVFLASH3 restores the exact layout, KVFLASH2 restores residency in identity block order. The expected chunk count, when given, is checked against the header before any state moves.
A test holds a KVFLASH3 blob captured from main's serializer (at 89fddc3d0, the geometry and seed are in the test). It loads, serializes back byte for byte, and equals what this pager writes for the same state (test_kvflash_pager_serde.cpp:1120, kvflash3_blob_written_by_main_loads_and_reserializes_byte_for_byte).
Left out of this PR
LUCE_KVFLASH_HOST_BACKING(kvflash_host_backing.h, its tests,set_host_backing). It has no caller until a later PR, so it ships with that caller. Here the pager keeps main'sset_pinned_host_backing(bool)(:205) and the singlecudaMallocHostpath.- Strided copies (one
cudaMemcpy2DAsync/cudaMemset2DAsyncper layer and K|V instead of one call per head). This is a performance change with no test need and no benchmark yet, so it is not here: the per-headcudaMemcpyAsync/cudaMemsetAsyncloops are main's, andgpu_runtime_compat.his untouched.
Callers that arrive in follow-ups
Used by a follow-up: row_base and the arena-space accessors, step pins, deferred free and complete, set_external_page_stream, set_batch_submit, page_out_checked, lru_victims, the reselect tail override, fill_slot_mask's valid_upto, and the KVFLASH2 parts. Nothing in src calls set_external_page_stream or set_batch_submit after this PR, and the unit tests do not exercise them.
The slice, fence, deferred-free and blob pieces are in one commit because they share the pager header and its tests.
Tests
test_server_unit, filtered to the two fixtures this PR changes: KvflashPagerSerde goes from 19 to 47 cases, KvflashPagerCudaRace from 3 to 5 (CUDA only). The new cases cover: the KVFLASH3 blob from main, KVFLASH2 parts and the streamed blob, the expected-chunk-count check, pinned page_out, step pins, deferred free, stale generation, row_base rows, attach refusal and bytes at the offset rows, the reselect tail override and ties, reselect() parity with main for the gemma4/laguna call shape, the truncate layout contract and identity relayout (bytes move, a swap cycle resolves), the bounded slot mask, and a byte-exact page-out/page-in round trip with a partial-chunk truncate on the CUDA side.
Full test_server_unit, passed / failed / skipped, main c5179e4b5 vs this branch. The difference is 30 on CUDA: 28 new KvflashPagerSerde cases and 2 new KvflashPagerCudaRace cases. It is 28 on HIP, where the CUDA-only race cases do not run.
| build | main c5179e4b5 | branch 25997804c |
|---|---|---|
| CUDA, RTX 3090 (sm86, Release) | 1224 / 0 / 6 | 1254 / 0 / 6 |
| HIP gfx1201 (hip:0) | 1207 / 0 / 4 | 1235 / 0 / 4 |
| HIP gfx1151 (hip:1) | 1207 / 0 / 4 | 1235 / 0 / 4 |
The GPU was idle before each run and one suite ran at a time. Each HIP log names its device (arch=gfx1201 on hip:0, arch=gfx1151 on hip:1).
Engine run
Qwen3.8-27B (Qwen3.8-27B-UD-IQ4_XS.gguf, md5 54db94b16b77e3fe024faa893cc4b80d), serial --kvflash 16384, two distinct prompts of 34583 and 37293 tokens, 128 max tokens, temperature 0, seed 1. The pool pages out on every arm, main and branch (pooled prefill: 34583 tokens through a 16384-token pool, and the same for 37293). Run on HIP only (gfx1201 and gfx1151); there is no CUDA engine run for this PR.
Setup: luce_server <model> --max-ctx 65536 --kvflash 16384 --no-kv-cache --cache-type-k q8_0 --cache-type-v q8_0, one server at a time, caches dropped before each arm, prompts sent one after the other. luce_server md5: main e01ec7db3b155947fc93c00f2acb4a07, branch 1b88cf5be5310ff36189e626291259fc. Arms run in the order main, branch, main repeat on each GPU. All requests show prefix_len=0 and disk_hit=false on all six arms, so no arm used a prefix or disk cache.
The prompts were answered in 61 and 61 tokens on gfx1201 and in 61 and 63 on gfx1151 (finish=stop), the same counts on every arm of a GPU. With two requests, median equals mean, and best and min are the two requests.
Speeds are from the server's chat DONE lines, tokens per second; wall in seconds (best = fastest). The log prints prefill time in steps of 0.1 s, so two runs with the same prefill time show the same prefill tok/s.
gfx1201 (hip:0):
| arm | prefill tok/s med / mean / best / min | decode tok/s med / mean / best / min | wall s med / mean / best / min | text vs main |
|---|---|---|---|---|
| main | 852.32 / 852.32 / 867.28 / 837.36 | 29.15 / 29.15 / 29.20 / 29.10 | 44.40 / 44.40 / 43.57 / 45.23 | - |
| branch | 852.32 / 852.32 / 867.28 / 837.36 | 29.15 / 29.15 / 29.20 / 29.10 | 44.41 / 44.41 / 43.58 / 45.24 | identical, both requests |
| main repeat | 851.31 / 851.31 / 867.28 / 835.34 | 29.15 / 29.15 / 29.20 / 29.10 | 44.42 / 44.42 / 43.61 / 45.23 | identical, both requests |
gfx1151 (hip:1):
| arm | prefill tok/s med / mean / best / min | decode tok/s med / mean / best / min | wall s med / mean / best / min | text vs main |
|---|---|---|---|---|
| main | 293.71 / 293.71 / 295.83 / 291.58 | 12.90 / 12.90 / 12.90 / 12.90 | 127.30 / 127.30 / 121.70 / 132.91 | - |
| branch | 290.66 / 290.66 / 291.10 / 290.22 | 12.90 / 12.90 / 12.90 / 12.90 | 128.56 / 128.56 / 123.63 / 133.50 | identical, both requests |
| main repeat | 291.13 / 291.13 / 291.59 / 290.67 | 12.90 / 12.90 / 12.90 / 12.90 | 128.34 / 128.34 / 123.41 / 133.27 | identical, both requests |
Branch vs the first main run, medians: on gfx1201 prefill 0.0%, decode 0.0%, wall +0.02%; main against its own repeat is -0.12%, 0.0% and +0.05%. On gfx1151 prefill -1.04%, decode 0.0%, wall +0.99%; main against its own repeat is -0.88%, 0.0% and +0.82%. On gfx1151 the first main run was the first after a server start and both later arms sit below it; against the main repeat the branch is -0.16% on prefill. This PR makes no speed claim and shows no speedup.
The tables above are for 1c0cbcdee. The review fix 25997804c was run again with the same harness, main (same binary) then branch on each GPU, with no main repeat. Branch luce_server md5 562a325f83bbf8d310ae34d146af6546. Text is identical to main on both requests on both GPUs, and identical to the 1c0cbcdee arms. prefix_len=0 and disk_hit=false on all four arms, and every arm logs pooled prefill for both prompts. Branch vs main, medians: gfx1201 prefill 858.44 vs 865.69 tok/s (-0.84%), decode 29.20 vs 29.30 (-0.34%), wall 44.07 vs 43.76 s (+0.71%); gfx1151 prefill 291.51 vs 292.72 (-0.41%), decode 12.90 vs 12.90 (0.0%), wall 128.24 vs 127.74 s (+0.39%). The same main binary gave 865.69 tok/s prefill on gfx1201 in this run and 852.32 in the run above.
The server has no logprobs, so near-tie margins between the two builds' token choices cannot be computed; text identity is the comparison.
Limits
- Decode speed rests on 61 to 63 generated tokens, two requests per arm. It is a small sample; the decode numbers above are not a decode benchmark.
- The engine run covers the serial
--kvflashpath on the two HIP devices. No CUDA engine run, and no run with a sliced pager (nothing in this PR creates one). - Three behaviour differences on the serial path when KVFlash is on:
page_outon a pinned chunk is refused.- A truncate to a prefix that fits the pool relays out to identity block order. The serial backends reset and rebuild on restore, so they did not see the old layout.
- On an exact score tie across the resident/host boundary,
reselectkeeps the resident chunk, where main's comparator left the order unspecified. The testreselect_score_ties_keep_resident_chunks_where_main_recalled_by_chunk_idrecords main's result for its case, a recall by chunk id.
- gemma4 and laguna call
reselect()with no arguments and never pin a step.reselect_without_pins_matches_main_for_gemma4_and_lagunareplays that call shape and compares events, residency and block order with what main's pager produced at 89fddc3d0. There is no gemma4 or laguna engine run. kvflash_require_base_zeroaborts the process on a sliced pager. It is unreachable in this PR.set_external_page_streamandset_batch_submithave no caller and no test here.- The strided-copy change and
LUCE_KVFLASH_HOST_BACKINGare not in this PR (see above).