perf(ds4): fuse expert-major MoE route combine
Summary
Speeds up DeepSeek-V4 heterogeneous prefill by fusing the post-MMID route reduction used by the production expert-major path.
The active path previously evaluated:
down_e -> mul(route weights) -> permute -> cont -> sum_rows -> add(shared expert)
This PR replaces the reduction and optional shared-expert add with one GGML_OP_DS4_MOE_COMBINE kernel. It now applies inside eval_moe_owner_expert_major_batched, so both the R9700 hot/shared owner and the Strix cold owner use it.
This is deliberately described as a post-MMID combine, not a down-projection epilogue: the grouped down projection still materializes down_e. The optimization removes the weighted-route intermediate and the following transpose/reduction chain.
Set DFLASH_MOE_FUSED_COMBINE=1 to enable the qualified expert-major path. Set it to 0 to retain the legacy graph sequence.
Heterogeneous prefill results
Measured on Lucebox6 with Radeon AI PRO R9700 gfx1201 + Strix Halo gfx1151, the ROCmFP2 model, unchanged production placement (hot/layer=60, hot stack 2580, cold stack 8428), caches disabled, and the same candidate binary with the feature forced OFF or ON.
Each cell used two warmups followed by five measured cache-cold requests. The order was OFF-A, ON-A, ON-B, OFF-B. All measured samples were retained.
| Workload | OFF pooled median ± MAD | ON pooled median ± MAD | Improvement | FFN median change |
|---|---|---|---|---|
401-token prefill, chunk=512 | 188.31 ± 0.45 tok/s | 198.23 ± 2.15 tok/s | +5.27% | 1626.75 → 1515.00 ms (−6.87%) |
2048-token prefill, chunk=2048 | 308.39 ± 6.08 tok/s | 339.93 ± 4.13 tok/s | +10.23% | 4119.10 → 3597.70 ms (−12.66%) |
Both bracketed comparisons agree in sign:
- 401: +6.09% in pair A and +5.00% in pair B; OFF bracket drift was +0.07%.
- 2048: +12.46% in pair A and +7.21% in pair B; OFF bracket drift was +3.90%.
The 2048 workload was noisier, but the gain survived both independent ON/OFF brackets. No outlier was deleted when calculating the pooled result.
Raw measured prefill samples
| Workload / cell | Five measured samples, tok/s | Cell median ± MAD |
|---|---|---|
| 401 OFF-A | 65.02, 63.17, 188.29, 188.92, 188.31 | 188.29 ± 0.63 |
| 401 ON-A | 51.87, 108.88, 200.38, 199.76, 200.38 | 199.76 ± 0.62 |
| 401 ON-B | 97.21, 89.41, 197.85, 198.61, 199.54 | 197.85 ± 1.69 |
| 401 OFF-B | 189.70, 188.93, 188.43, 188.30, 188.03 | 188.43 ± 0.40 |
| 2048 OFF-A | 298.83, 303.71, 304.37, 303.99, 300.13 | 303.71 ± 0.66 |
| 2048 ON-A | 341.56, 267.85, 342.75, 342.83, 305.69 | 341.56 ± 1.27 |
| 2048 ON-B | 311.90, 334.57, 338.29, 351.93, 341.82 | 338.29 ± 3.72 |
| 2048 OFF-B | 315.55, 312.40, 313.37, 322.52, 330.52 | 315.55 ± 3.15 |
The tested server-code base was b9d37b32. Current main at publication is f5475131; the intervening commits only change documentation and assets, so the server code under test is unchanged. The replacement PR commit is based directly on f5475131.
Kernel-level measurement
The focused benchmark uses the model's exact hidden width (n_embd=4096) and top-k 6. It compares the actual legacy graph with the fused operation after two warmups and seven measured samples.
| Device | Tokens | Legacy median | Fused median | Combine speedup |
|---|---|---|---|---|
R9700 gfx1201 | 401 | 0.761 ms | 0.055 ms | 13.76× |
R9700 gfx1201 | 2048 | 4.522 ms | 0.467 ms | 9.68× |
Strix gfx1151 | 401 | 2.031 ms | 0.239 ms | 8.50× |
Strix gfx1151 | 2048 | 11.023 ms | 1.163 ms | 9.48× |
These are combine-kernel speedups, not whole-model speedups. The end-to-end gains are the pooled figures above.
Correctness and execution proof
- Production telemetry recorded exactly 86 fused launches for one 401-token prefill: 43 hot/shared-owner launches and 43 cold/no-shared-owner launches.
- Every strict timing request matched the expected output SHA-256
cd5cb9fb5ac3c4f4007e8b41d117da21622439cd05c1728f3e82f90e4f869dad. - The direct combine differential covers top-k 4/6, token counts 1/3/33/401, shared and no-shared outputs, actual legacy-vs-fused CPU/HIP byte parity, masked routes with NaN-poisoned omitted lanes, and signed-zero parity. It passed on both
gfx1201andgfx1151. test_deepseek4_mmid_grouped_cuda: 72/72 cases passed on each GPU, including masked fused-zero coverage.test_deepseek4_unit: passed on each GPU.git diff --check: clean.- No foreign KFD process was observed in any retained strict cell; KFD was clean after qualification.
- A fresh independent review of the final replacement diff found no remaining correctness issue after the signed-zero and legacy-differential fixes.
The qualified evidence is preserved on Lucebox6 at:
/home/cheese/pr658-active-path-final-r2-20260826T1624Z/qualified-evidence
Manifest SHA-256: 44f1a353b6d84cd67f8339796296537b7ee7e6b901b57b8b9553d4cb89a5639f
Scope
This PR claims a heterogeneous prefill improvement. It does not claim a decode speedup. A deeper fusion that avoids materializing down_e would require a separate grouped-MMID down-projection redesign and is outside this change.