[WebGPU EP] Fuse QMoE 1-token decode path to reduce GPU dispatches
Description:
Summary
Fuse the QMoE 1-token decode path to reduce GPU dispatches from 17 (1 + k×4) to 5 (gate + fc1 + swiglu + fc2 + mix), improving token generation throughput by ~21% on Meteor Lake for the gpt-oss-20b MoE model (19 → 23 tps).
Motivation
The QMoE operator processes Mixture-of-Experts layers by selecting top-k experts (k=4) per token. In the original 1-token decode path, each expert is processed serially with 4 dispatches (gather + fc1 + swiglu + fc2 + mix), totaling 17 GPU dispatches per QMoE call. Since each dispatch has M=1, the GPU is underutilized and CPU dispatch overhead dominates.
Approach
For the 1-token path (num_rows == 1):
Gate1Token — Select top-k experts and output an indirect_experts buffer mapping row index → expert index
Batched fc1 MatMulNBits — Run a single M=k matmul with per_row_weight_indirect mode, where each row selects a different expert's weights via the indirect buffer
SwiGLU — Apply activation on all k rows at once
Batched fc2 MatMulNBits — Same per-row expert selection for the down projection
FusedFinalMix — Accumulate all k weighted expert results into the output
Follow-ups
Fuse Batched fc1 MatMulNBits + SwiGLU
Fuse Batched fc2 MatMulNBits + FusedFinalMix
Finally, we only need three shaders: Gate1Token, fused Batched fc1 MatMulNBits, fused batched fc2 MatMulNBits.