perf(qwen35): fuse the DFlash2 concurrent verify Gated DeltaNet route

Merged
GR
1mo ago
+689-44

Summary

Reduces kernel launches and temporary memory traffic during Qwen3.5-family DFlash2 concurrent verification. The previous graph separately prepared gates, gathered convolution and recurrent state, and wrote recurrent states that the accepted-prefix commit never consumed.

This change fuses that preparation into the existing convolution and recurrence execution. Verification reads each sequence's persistent state through its slot mapping and produces the attention output and replay data needed to commit accepted tokens.

Optimizations

  • Fused convolution preparation. ggml_ssm_conv_tree_step reads the mapped convolution history, combines it with the current projected tokens, follows the tree parents, and applies convolution plus SiLU in one kernel. It also produces the window used by the later commit. This replaces the history gather, transpose/concatenation preparation, and separate activation.
  • Read-only mapped recurrence. ggml_gated_delta_net_mapped_verify reads recurrent state directly from the owning slot. It allocates and writes attention output and the captured replay log, omitting the gathered base-state copy, final-state output, and per-token intermediate states that this route does not consume.
  • Gate preparation inside the recurrence. Strided views of the stacked beta/alpha projection feed the kernel directly. It applies sigmoid and softplus with the bias and scale, removing the two contiguity copies and separate gate operations.

The convolution preserves tap and accumulation order, and the gate expressions match the existing operations. Persistent convolution and recurrent state remain read-only during verification; the existing replay-log commit advances them after acceptance is known.

Scope and fallback

build_delta_net_block selects the fused operations only for the mapped concurrent tree segment that commits through the replay log. Dense-chain execution, prefill, SpecLA, and single-sequence tree verification retain their existing selection paths. The new operations support CUDA/HIP, and DFLASH_QWEN35_NO_FUSED_KERNELS=1 restores the unfused graph.

Validation

The focused GPU test compares attention output, replay logs, convolution output, and commit windows bit for bit across four shapes, including grouped and non-grouped recurrence. It also verifies that persistent state remains unchanged. The test passes on R9700 and Strix Halo.

Real Qwen3.8/DFlash2 runs with fusion disabled and enabled preserve the checked outputs and pass mixed-length concurrency, cancellation, and slot-reuse checks. Existing measurements show higher throughput across one to four clients; single-client p95 time to first token increased 1.99%, so this is not a latency improvement at every concurrency level.