perf(qwen35): recover dense AR replay fast path

Draft
DU
3mo ago
+81-23

Summary

Extract the server-only dense AR replay micro-optimizations from the replay-win branch into a focused PR that does not depend on ggml PR24 or any submodule bump.

This keeps the in-place GDN/submodule work isolated in #473, while letting reviewers evaluate the independent Qwen35 AR decode fast path now.

What changed

  • Fuse DeltaNet gated output with ggml_swiglu_split.
  • Skip graph-level FWHT K rotation for q4_0 by default, while preserving explicit overrides:
    • DFLASH_FORCE_WHT=1 restores the old graph-level rotation.
    • DFLASH_NO_WHT disables it.
  • Skip Q/K repeat-back in pure AR where the shape is already single-token decode.
  • Use contiguous concat inputs / materialized qkv_T to reduce concat-cont overhead.
  • Reuse the Qwen35 AR decode graph inside the same 256-token FA bucket so CUDA graph warmup/replay is not reset every token.

Scope and dependencies

  • Server-only: Qwen35 backend/graph code and one small graph-options helper header.
  • No server/deps/llama.cpp submodule change.
  • No ggml_cuda_set_skip_props_check / props-check hook.
  • No in-place GDN API use; that remains in #473 and depends on ggml PR24.
  • Does not include the KVFlash library/integration work from #466/#468/#469/#470.

Why

Our best 6-turn dense replay result recovered a decode-rate win over llama.cpp, but #473 mixed those micro-opts with the ggml-dependent in-place GDN consumer. Splitting this lets the low-risk server-side perf work land independently and keeps the reviewer surface small.

Latest measured replay context from the winning stack:

  • llama.cpp slot-cache: prefill about 6.900s, decode 5.394s, wall 12.294s, weighted decode 42.642 tok/s.
  • Luce patched/final stack: prefill about 8.1-8.7s, decode about 5.28-5.40s, wall about 13.6-14.1s, weighted decode about 43.6 tok/s.

This PR carries the server-only part of that decode recovery. Prefill parity remains separate work.

Validation

Local focused worktree, upstream ggml submodule a317b0ea0fc9eb716a311976fed8dc0f301dc09f:

  • env CC=/usr/bin/gcc-11 CXX=/usr/bin/g++-11 CUDACXX=/usr/local/cuda-12.6/bin/nvcc LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64:/usr/lib/wsl/lib cmake -S server -B server/build-ar-replay-microopts-gcc11 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-11 -DGGML_CCACHE=OFF
  • env LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64:/usr/lib/wsl/lib cmake --build server/build-ar-replay-microopts-gcc11 --target dflash_server test_server_unit -j 8
  • env LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64:/usr/lib/wsl/lib server/build-ar-replay-microopts-gcc11/test_server_unit
  • Result: 2022 assertions, 0 failures

Notes:

  • A first configure with /usr/bin/nvcc 12.0 + GCC 13 failed at CUDA compiler identification due the known glibc _Float* incompatibility; CUDA 12.6 + GCC 11 configured and built successfully.
  • The build disables optional BSA because that optional submodule was not initialized in this focused worktree.