perf(ds4): batch heterogeneous MoE prefill by expert
Summary
Accelerates DeepSeek-V4 heterogeneous prefill by batching owner-local routed experts into grouped ggml_mul_mat_id graphs instead of issuing a serial graph for every active expert.
- enables the expert-major heterogeneous path from 32 tokens instead of 512;
- builds grouped gate/up/down MMID graphs per owner GPU;
- adds GPU-native route histogram, prefix, and scatter work for up to 256 experts;
- preserves masked owner routes as exact zero contributions;
- retains
DFLASH_MOE_GROUPED_MMID_PREFILL=0andDFLASH_MOE_EXPERT_MAJOR_MIN_TOKENS=512as rollback controls.
Classification: primarily a performance PR. It also contains the masked-route correctness fix discovered during qualification, but that fix supports the optimization rather than changing the PR's main purpose.
Fresh main-snapshot requalification
Freshly remeasured on Lucebox6 against recorded main snapshot f686c447f067a04ea100a996e4c826e8cc4decc1, using the same ROCmFP2 model, placement, chunk size, cache-cold requests, two warmups, and five measured samples per cell.
| Workload | Recorded main | #640 | Delta |
|---|---|---|---|
401-token prefill, chunk=512 | 94.59 tok/s (MAD 0.15) | 178.98 tok/s (MAD 0.17) | +89.2% (1.89x) |
2048-token prefill, chunk=2048 | 329.16 tok/s (MAD 3.23) | 298.97 tok/s (MAD 2.05) | -9.17% |
The current positive result is the short/medium 401-token workload: 1.89x the recorded-main-snapshot throughput. #640 alone is not a current wide-context win; at 2048 tokens it regresses. The stacked #647 scheduling PR recovers that wide result while adding another 7.0% at 401 tokens.
All retained responses produced SHA-256 cd5cb9fb5ac3c4f4007e8b41d117da21622439cd05c1728f3e82f90e4f869dad, and every retained request was cache-cold.
Evidence:
- 401-token root:
/home/cheese/pr640-pr647-fusion-live-main-short401-20260825T121700Z - 401 manifest SHA-256:
6a9d6b586ac04fc153b9af2b87a5175c01fe3178d9808c9817f7b89f87ac8d21 - 2048-token root:
/home/cheese/pr640-pr647-fusion-live-main-requal2-20260825T112300Z - 2048 manifest SHA-256:
b73afed6803e26ed111f521eb0690cf9d8056e30c5f890b0067a5604d181e66a
Original matched qualification
The original same-session qualification against main at ac22a3ed measured:
| Workload | Main | #640 | Result |
|---|---|---|---|
401-token prefill, chunk=512 | 54.16 tok/s | 184.11 tok/s | 3.40x (+239.9%) |
| MoE FFN time, 401 tokens | 6,791.6 ms | 1,661.2 ms | 4.09x reduction |
2048-token prefill, chunk=2048 | 292.42 tok/s | 304.69 tok/s | +4.2% |
These remain valid historical matched-session results for those exact heads and machine state. The fresh table above is the relevant comparison to the newer recorded main snapshot; absolute rates must not be mixed across sessions.
Correctness and validation
- exact full-model output parity across the retained 401- and 2048-token cells;
test_deepseek4_mmid_grouped_cuda: PASS across 72 cases on physical gfx1201 and gfx1151, including wide masked ROCmFP2/ROCmFP3 routes;test_deepseek4_unit: PASS on physical gfx1201 and gfx1151;- compact MMID destinations are zero-initialized before dispatch so unwritten masked lanes cannot contribute stale or NaN values;
git diff --check: clean;- all currently reported GitHub checks: successful; GitHub reports the PR mergeable and clean.