perf(qwen35): use in-place GDN state write in pure AR
Summary
Dense (27B) qwen35 AR-decode performance stack: in-place GDN state write, 256-token FA-bucket CUDA-graph reuse (with a props-check skip), FWHT K-rotation gate for q4_0 KV, swiglu fuse, and concat fast-paths. Rebased clean onto current main.
Supersedes #476, whose micro-opts are a subset of this stack.
Dependency (resolved)
The in-place GDN op itself (ggml_gated_delta_net_inplace, originally Lucebox ggml #24/#27) is now vendored into main via #499 (5e302cbb, "port(ggml): in-place GDN state write + qwen dense decode recovery"). main ships the op but still calls the non-inplace variant at the qwen call site (qwen35_target_graph.cpp:941); this PR wires the qwen35 dense pure-AR consumer to use it (ce2325aa). After the ggml-only vendorize (#486), there is no external submodule bump left — the earlier "draft until the ggml submodule commit is available" blocker no longer applies. This branch was rebased onto current main, dropping the obsolete submodule-gitlink bumps.
Commits
ce2325aause in-place GDN state write in pure ARe1ce9c73fuse silu+mul → swiglu_split in DeltaNet gated output96fd3ea9gate FWHT rotation for q4_0 KV + skip repeat_back in pure AR5fd7c3fbcont concat inputs for fast concat_cont patha5915e29graph reuse in AR decode — skip rebuild within 256-token FA bucketa8253730reduce concat cont overhead — materialize qkv_T once9d450624recover dense AR replay fast path
Correctness (verified)
- Adversarial static review (Codex): clean — no silent-corruption, race, or aliasing across all 7 commits. Checked with file:line: 256-bucket boundary condition, in-place GDN aliasing, FWHT K/Q rotation symmetry, swiglu numerical equivalence.
- AR-replay bit-identity:
DFLASH_AR_NO_REUSEON vs OFF → byte-identical greedy output at 4K and 16K (16K crosses two bucket boundaries). The 256-bucket rebuild guard re-captures exactly when the FA window advances; the props-check skip is a launch-overhead optimization only, not a numerics change. - In-place GDN bit-identity: revert-
ce2325aaA/B with WHT held constant → byte-identical. Output-inert when live. - FWHT-skip fidelity: teacher-forced over 32K of real corpus, bucketed vs an f16 reference — top-1 agreement gap ≤ 0.6pp (flat with depth) and PPL flips direction bucket-to-bucket → within noise. Skipping the K-rotation on q4_0 does not measurably degrade quality on real text.
Performance
27B dense, q4_0 KV, pure AR, temp 0, HTTP serving path, median of N post-warmup, same binary both arms (accept_rate=0, prefix_len=0 verified — no caching asymmetry). AR graph-reuse, replay ON vs OFF:
| context | replay ON (tok/s) | replay OFF (tok/s) | Δ decode |
|---|---|---|---|
| 3.1K | 41.25 | 40.70 | +1.35% |
| 14.1K | 38.45 | 37.75 | +1.85% |
| 30.6K | 34.90 | 34.35 | +1.60% |
| 71K | 29.15 | 28.70 | +1.57% |
| 114K | 24.55 | 24.10 | +1.87% |
The graph-reuse decode gap is a flat ~1.6% band across a 37× context range (3K→114K) and never clears each arm's own run-to-run spread (e.g. at 71K, ON's own min–max is 2.7% vs a 1.57% gap). It does not grow with depth, contradicting the "launch overhead scales with KV length → win grows with context" rationale. prefill throughput and wall are statistically identical between arms (the reuse only touches per-token decode launch).
The optimization's originally reported ~+6% (106→113 tok/s) was measured on Q3_K_XL at 16K; it does not reproduce on Q4_K_M at any depth tested here (3K/16K/32K/71K/114K). Read this as: correctness-neutral and safe on Q4_K_M, but no HTTP-measurable decode win on this quant — the speedup appears quant-specific.
Notes
- Rebase resolved the stale
llama.cppsubmodule-gitlink vs the ggml vendorize onmain(dropped the obsolete gitlink bumps;rererecombined main'ssrc[7]persist-wiring with thepure_arin-place op call — verified bit-identical). - Kept as draft pending review.