PR #664 - Luce-Org/lucebox

Merge-ready update — 2026-09-08

Current head: 13648dba4528ad592f85a766cde4cfc0ff6fb581. Main through 561a7e9d0 (#716 DS4 disk-prefix-cache integration) is merged without dropping its snapshot or disk-cache changes. The subsequent main change, #719, touches only README.md; GitHub still reports no merge conflicts. Cubic review is complete, with all discussions resolved after inspection; this is not a formal Cubic approval. Code review is complete, with the hardware-CI exceptions below explicitly retained. No merge has been performed.

Fixed cached verifier raw-row capture: runtime read indices now track ring-write indices on every replay. Sparse attention, native F16 KV, packed small-width indexing, adaptive verification, and existing opt-in defaults remain intact.
Added a regression that fails 48/72 cases before the fix and passes 72/72 after, covering widths 2–5, F16/F32, ring wraps, and positions through 128K. Existing saved-row attention 80/80 and RoPE replay 20/20 also pass.
Fixed per-token indexer mask alignment when the scorer skips a query prefix: 1/5 before → 5/5 after exact offset/layout cases. PFlash now accepts finite [0,1] ratios and rejects NaN/infinities; zero and NaN regressions are reproduced and fixed, with eight admission cases passing.
Clean lb8/gfx1151 build with ROCm 7.2.4 and HIP graphs enabled. DS4 suite (including the new disk-snapshot tests), server 453/453, feature gate, sequence-engine 15, batch-plan 62, and paged-KV 25/25 checks pass. The two-device HC test skips with only Strix exposed.
New-head live lb8 smoke: strict 7,680-token retrieval 5/5, 59 output tokens, matching known-good response SHA256 df780faf7676ccc768737cb23d0534555c6e4785382a7f225d4842e82dcffb87. Adaptive DSpark (cap 4), all six experts, sparse F16 fused verification active, prefix/PFlash caches off. One cold request with chunk 512 is correctness evidence, not a matched throughput comparison.
Corrected the sparse-verifier documentation to distinguish single-lane/ratio-4 selection from other batched layouts. Its example export is now commented out and visibly experimental. PFlash has its own section; legacy daemon GPU/residency limits are explicit and covered by a parser contract test.
Reviewed and resolved discussions with evidence. Intentional lazy PFlash reload and legacy wire limits are documented, and the Qwen35-only memory heuristic / expanded negative-test / two performance suggestions are non-blocking follow-ups, not claimed implemented. Cubic's final two comments repeat the saved-row rule despite the ratio-4 qualification and misattribute a neighboring helper comment; code-backed dispositions are posted. The RTX3090 runner is confirmed offline.
CI passes on this head: Linux, Windows, Strix Halo, DGX GB10, workspace. R9700 passed the DS4 suite and 34 other selected suites, but its global free-VRAM leak check failed under an unrelated 24.5 GB / 100%-busy model workload. The unchanged registry test passes 5/5 on lb8's idle physical R9700, with zero-byte delta every run. This host-side HIP allocation test does not launch gfx1151 numerical kernels on R9700. The CI job remains red, not relabeled; GitHub refuses a scoped rerun while the parent workflow is active. No check or threshold was weakened.

Performance boundaries: previous-head 8K off/on medians were 30.05 → 33.00 tok/s decode; these are not measurements of this new head. The 123K off-profile baseline and subsequent guarded correctness attempts produced no valid result because of swapping. The final isolated request stopped at ~1.066 GiB cumulative swap-out with ~10.53 GiB still available, so no OOM diagnosis or long-context correctness/throughput result is claimed. No fresh long-context gain, universal byte-identity guarantee, or Vulkan-parity claim is made. All temporary test servers are stopped.

New CI run · Final review-fix evidence

Previous cached-RoPE fix — 2026-09-08

Previous head: cbeea1f78925578be37e3d6b8d00e37378294786. The tested sparse-verification fix is now pushed; fresh CI and Cubic review are required. This is not a merge or performance-parity declaration.

Fused inverse RoPE now reads runtime token positions during graph replay. Cached forward-Q RoPE remains standalone to retain the validated verifier output; sparse flash attention, native F16 KV, adaptive DSpark, and uncached prefill fusion remain enabled.
Exact executable sources were validated on lb8 (gfx1151, ROCm 7.2.4, HIP graphs on): 20/20 new replay cases, 80/80 saved-row cases, 451/451 server tests, feature gate, and two unchanged 7,680-token adaptive sparse F16 retrieval requests. Both responses match the known-good output byte for byte. This does not imply universal byte identity.
No profiling experiments or diagnostic switches are included. Long-context speed retention and matched normal/ROCprof validation remain pending; no fresh Vulkan-win claim is made.

Prior-head review — 2026-09-07

Previously reviewed head: 059086abbd7a80ce84ab4e048aeabcc97f089eb4. Main was integrated at that revision. This PR does not include changes unique to #667. The CI and review results in this section predate the new fix.

All four original correctness findings are fixed and have regression coverage: saved raw-row visibility, synchronization before parking, failed drafter-load cleanup, and rejection of unsupported DS4 split/PFlash configurations. Current main's paged/PFlash guard is preserved.
CI passed on that head: Strix Halo, R9700, DGX GB10, Linux build, Windows build, and workspace import smoke.
RTX 3090 GPU CI and speed profile fail before building because the NVIDIA driver is unavailable. These are acknowledged infrastructure exceptions, not test passes; no CI checks were disabled or bypassed.
Cubic completed its full prior-head review. Its two remaining P3 comments are performance follow-ups, with code-backed dispositions in their threads: heterogeneous direct-API compression batching and a smaller near-512 sort network. Neither is claimed implemented.

Rollout boundary

Keep DFLASH_DS4_SPARSE_DECODE_FLASH=0 (the default is off) until broader workload and performance qualification. The sparse example in server/docs/DS4.md remains an experimental opt-in. The prior head failed strict five-line formatting with sparse F16/F32 verification. The new cached-RoPE fix resolves that reproduced failure in two repeated F16 adaptive runs; it does not establish universal output identity or qualify every context/checkpoint.

AR and adaptive DSpark q<=4 with fused explicit attention both pass the same 7,680-token strict retrieval task and produce identical response hashes. This single-task result is not a universal byte-identity guarantee. PFlash is lossy, optional, and supported for monolithic serving only; do not enable it for true-context or exact-retrieval performance comparisons.

Previous full-suite verification (`059086abb`)

Clean ROCm 7.2.4 / gfx1151 build with HIP graphs enabled; 451/451 server tests; 80/80 saved-row GPU cases (q=1..5, 2K/30K compressed rows, F16/F32, mask/direct selector); feature-gate, ROCm-format, packed-indexer, dispatch and fused-combine checks; 8/8 paged-attention checks. The two-device HC subtest skips with only Strix exposed. No warning-free claim.

Final real-model PFlash/FlowKV lifecycle smoke: two successful HTTP requests with correct short answers, 2,578 target tokens → 1,288 effective tokens, one target park/restore cycle, and one compression-content cache hit on the second turn. This used exact prefill and request-scoped drafter residency. The initial harness assumed the whole-prompt log format and two fresh compressions; its post-run assertion was corrected by separately validating the actual multi-turn FlowKV route against both saved responses and lifecycle logs. No kernel/server failure occurred, and the temporary server was stopped afterward.

CI run · Matched runtime controls

Changes

Add opt-in sparse fused verifier attention with direct F16 KV at long context.
Add packed small-CM rocWMMA indexer dispatch for widths 2–5 on RDNA3.5/RDNA4; qualified bit-identical to the generic indexer.
Add DS4 PFlash with Qwen3-0.6B, cross-tokenizer conversion and drafter residency lifecycle.
Correct batched verifier visibility masks and preserve overwritten raw rows.

Historical measurements — before the current main integration

The original branch reported 122,879 prompt tokens, 256 output tokens and six experts: client decode 17.513 → 22.513 tok/s (+28.6%), unchanged output SHA, zero swap. The cited Vulkan reference was 32.60 tok/s. These are historical results, not remeasured current-head performance and not a parity claim.

The original PFlash smoke reported 8,192 target tokens → 1,739 effective tokens with 2.18 s Qwen scoring. Compression changes effective context and must not be used to claim matched-context speed.