#667 perf(ds4): accelerate ROCm decode and long-context selection

Summary

Reconciled with main after #664 and #713 merged. This PR accelerates DeepSeek V4 Flash verification on gfx1151 without reducing the model's six routed experts. Here, q=4 means seed-inclusive speculative verification width four, not four experts.

Current head: 901fd3584. Latest lb8 qualification includes main/#717 and
passes 472/472 server tests plus the DS4/ROCmFP GPU suites. Fresh CI/review
are pending; the Strix failure on the previous head was investigated and fixed
as described below. No merge or auto-merge has been performed.

Merge qualification — 2026-09-08

Integration commit d214d4205 merges main 04451f456 (including #664 and #713).
All seven conflicts are resolved. Sparse verification remains opt-in; main's
MMVF dispatch is preserved exactly. The obsolete PR-specific q4 override is gone.

All nine previous review items are addressed. In particular, the expanded FP3
fixed-level parity test exposed 5/192 mismatches after the initial arithmetic
rewrite, while the same fixture passed with main's library. Fixed levels now
retain the original fold; learned-codebook pairing and FP2 row4 remain active.
The strict test now passes. No tolerance was relaxed.

lb8 checks on d214d4205 (ROCm 7.2.4, gfx1151, Release, HIP graphs ON):

• Clean server/focused builds; server suite 461/461; feature gate passed.
• Complete DS4 suite passed with split-KV enabled and disabled, including
  #664 saved-row replay 72/72, attention 80/80, RoPE 20/20, and #713 rollback
  on CPU and HIP sync/async/pinned.
• 12,714 raw-ring mask cases; failed init preserves sparse opt-in; top-k exact
  selected-set parity and padding ties across ten tile sizes: passed.
• FP2 mixed-codebook GLU widths 1–5 and row4-off fallback, FP3 fixed/learned
  per-slice exact parity, ROCmFPX MMQ, and allocation registry checks: passed.
• AR and adaptive DSpark 8K model smokes passed, both 5/5 retrieval and the
  same expected response SHA256. All six experts; no prompt compression/cache.
  The speculative smoke forced the incremental mask path on.
• FP2 fused q3–5 takes ~17% less kernel time than current main, with matching
  hashes. All ten FP2/FP3 q1–5 benchmark hashes match. FP3 timings are roughly flat
  (q1 sample is 3.3% slower); this is not an every-kernel speedup claim.

These are merge/correctness checks, not a new 128K or Vulkan comparison.
The retrieval fixture has high acceptance and stayed q4 despite adaptive being
enabled; changing widths/rejections are covered by component tests. Other model
families were compiled/unit-tested, not live-qualified here.

Artifacts: /data/pr667-ready-20260908.k8ieiO/ on lb8.
That revision passed Linux/Windows builds, workspace, DGX, Strix and R9700 CI;
RTX3090 was cancelled. See the follow-up below for the current head's status.
Historical workflow measurements below are retained with their original revisions.

Review fixes — 305eb5099

Six new review findings fixed; the remaining finding was an early-init test
false positive, explained with the old call order and clarified in the test.
All seven review threads have individual evidence replies.

• Qwen MoE's shared opt-in controller can back off to two rows; the legacy
  default and explicit fixed-width override are preserved.
• Shared Qwen chain/tree acceptance accounting includes matching seed semantics,
  actual tree nodes (not padding), remaining-budget limits, and all exit paths.
• Split-KV controls now honor empty/0 values and kill-switch precedence.
• Sparse-threshold wording and the five GGML inventory entries are corrected.

Exact-head lb8 Release build and 471/471 server tests passed, plus the full
DS4 GPU suite with default flags, all split-KV flags =0, and the disable alias
overriding force=1. FP2 GLU q1–5 and row4-off, FP3 fixed/learned slice parity,
ROCmFPX MMQ, and gfx1151/gfx1201 registry checks all passed again.
18 focused local tests passed, including a 512-combination flag matrix that
fails against the old predicate. No DS4 kernel arithmetic changed in this fix.
These checks are not new end-to-end throughput measurements.

Logs: /data/pr667-ready-20260908.k8ieiO/review-fixes/.
CI: https://github.com/Luce-Org/lucebox/actions/runs/34238905547
subsequently exposed the intermittent Strix failure fixed below. Cubic also
requested the same final-budget cap for tree telemetry; that is now fixed.
Main has since added
#717; the conflict-free integration with 27bc79651 also passed the lb8 build,
471/471 server tests and the same complete GPU regressions. Its separate logs
are under review-fixes-main717/. The 18 local tests also passed ASan/UBSan.
Not merged; auto-merge remains disabled.

Final correctness fixes — e03cdfaf7, 901fd3584

The fresh Strix CI job was not an infrastructure failure: saved-row attention
failed at q5 / 30,720 compressed rows / F16 KV. Inspection found a shared-memory
race: a fast wave could overwrite the reduced maximum with its partial sum
before other waves had read the maximum. Added the missing read-before-reuse
barrier in split-KV and the two grouped kernels using the same scratch pattern.

A deliberately staggered-wave diagnostic using the exact failing shape exited
1 before the barrier and 0 after it. The diagnostic delay/entry point is removed
from production. The unchanged 1e-6 assertion is now also exercised through
128 repeated launches with refreshed graph inputs. No optimized path was disabled.

Tree acceptance now also caps final-step offers by the remaining output budget,
with regression cases for three-token and root-only final steps.

Final lb8 Release build + integration with main 27bc79651:

• 472/472 server tests, feature gate, and 19 focused ASan/UBSan tests pass.
• Full DS4 GPU suite, including saved-row replay stress, passes with defaults,
  all split-KV flags =0, and force=1 overridden by the disable alias.
• FP2 GLU widths 1–5/row4-off, FP3 fixed/learned exact slice parity, ROCmFPX MMQ,
  and gfx1151/gfx1201 registry checks pass.

Logs: /data/pr667-ready-20260908.k8ieiO/final-race-fix/; controlled before/after
diagnostic receipts: race-probe/target-before.log and target-after.log.
Fresh CI: https://github.com/Luce-Org/lucebox/actions/runs/34241514672 .
Await its results and the new Cubic review before giving a final merge green light.

Cleaned scope

• Preserve main's broader narrow-F16 MMVF dispatch and operator overrides; remove the superseded PR-specific q4 switch.
• Reuse activations across four ROCmFP2 output rows and tune ROCmFP2/3 Wave32 loads/launches.
• Update large verifier masks by device-clear plus negative ranges instead of transferring the full mask.
• Split indexed MLA KV work across more Wave32 blocks, then combine online-softmax state.
• Replace full long-context top-k sorting with an exact two-stage block-radix selector.
• Preserve #664's explicit opt-in sparse verifier-attention policy. A requested but absent drafter must not change AR attention globally.
• Keep the shared adaptive-width controller available for linear-chain drafters, but opt-in. Fixed width is the production default.

The draft-only fused MoE combine and direct-visible attention experiments were removed: neither cleared the qualification threshold, and the latter's shape contract was not broad enough for merge.

Historical real-workflow A/B (before the main reconciliation)

Matched on lucebox8's Strix Halo (gfx1151), ROCm 7.2.4, the same target and DSpark GGUFs, q4 verification, six routed experts, q4 KV, sparse prefill, and 32K server context. Sparse verifier attention was explicitly enabled on the exact base (5452b4a) to match the head (5eb4fbe) default.

Workflow	Exact base	This PR	Change	Quality
HumanEval	27.33 tok/s	35.77 tok/s	+30.9%	2/2 -> 2/2
GSM8K	24.77 tok/s	32.28 tok/s	+30.3%	2/2 -> 2/2
Agent, 2K/8K/24K prompts	22.29 tok/s	30.09 tok/s	+35.0%	6/6 successful in both

The two 24K agent cases averaged 22.42 -> 29.71 tok/s (+32.5%). Prefill was essentially unchanged: HumanEval 43.5 -> 44.7 tok/s, GSM8K 31.3 -> 32.5 tok/s, and agent 94.1 -> 93.9 tok/s. This PR closes a substantial decode gap; it does not claim to close the remaining raw-prefill gap to Vulkan.

On matched requests, verifier time fell from roughly 125-133 ms/step to 84-101 ms/step. Benchmark artifacts and logs are retained under /data/pr667-merge-ready/ on lucebox8.

The sparse/quantized path is not an end-to-end byte-identity mode, so task correctness is the workflow quality gate. These historical results are not a fresh comparison against current main or Vulkan.

Historical adaptive width result

Adaptive width remains opt-in via DFLASH_ADAPTIVE_SPEC_WIDTH=1 or the DS4-specific DFLASH_DS4_ADAPTIVE_WIDTH=1. On the same real workflows it was slower than fixed q4: HumanEval -4.7%, GSM8K -15.9%, and agent -3.5%. The implementation is retained for future backend-specific qualification, while all non-DS4 backends preserve their previous default policies.

Historical verification (old head 5eb4fbe)

• Release dflash_server build on ROCm/gfx1151.
• test_server_unit: 431/431 passed after final cleanup.
• deepseek4_unit, including long-context top-k and indexed-attention GPU coverage: passed.
• test_rocmfp_mix_gateup_glu and test_rocmfpx_mmq: all parity cases passed.
• Exact block-radix selector: 2.36-4.55x faster than full sort across DS4-relevant sizes, exact selected set for every case.
• Real target tokenizer harness: passed.
• Codex Responses API and Open WebUI streaming probes: passed.
• git diff --check: clean.

An unrelated all-target HIP build still hits the pre-existing cudaErrorNoDevice compatibility issue in unchanged test_rms_norm_hip.cpp; the production server and all PR-focused targets build cleanly.

Controls

• DFLASH_DS4_SPARSE_DECODE_FLASH=1: explicitly opt into experimental single-HIP sparse verifier attention; unset/0 keeps explicit attention.
• GGML_CUDA_MLA_NO_SPLIT_KV=1: disable split-KV MLA.
• GGML_DS4_TOPK_BLOCK_RADIX=0: restore hipCUB full sort.
• DFLASH_ROCMFP2_ROW4=0: restore the two-row ROCmFP2 kernel.
• DFLASH_DS4_INCREMENTAL_VERIFY_MASK=0: restore full verifier-mask transfer.