feat(qwen4exp): MTP speculative decoding (stacked on #774)
Stacked on #774 (Luce-Org/lucebox, feat/qwen4exp-strix-halo). This PR only contains the MTP work on top of it.
What
Speculative decoding for Qwen3.8-Flash-Next with the model's own MTP block. The single MTP layer is reused to chain up to seven drafts per step, and the trunk verifies them in one batched forward.
- The sidecar (
MTP/mtp-*.ggufnext to the model) is found automatically. Without it, decode runs without MTP. --verify-width 0(default): adaptive width, up to k=7 drafts, from per-depth acceptance and a per-request cycle-cost fit.--verify-width 1: MTP off.--verify-width 2..8: fixed k=1..7.- Drafts are cheap: the draft head covers 64,000 vocabulary rows (control/special tokens plus the lowest ids), picks its token on the GPU and feeds the next draft from device memory; catch-up runs attention/FFN only for the last retained row. Drafts are proposals only, so this never changes output.
- No environment variables.
- Also includes the QSA decode speed commits this work was built on: fused block sort + cell expansion, and decode-graph reuse across pooling steps and KV buckets.
Exactness
A draft is accepted only if it equals the token the trunk samples, so output is byte-identical to MTP off for the same seed. Rollback restores the KV/recurrent state bit-exactly.
test_qwen4exp_mtp: 376 acceptance/rollback cases.test_qwen4exp_mtp_dispatch: 21 cases.smoke_qwen4exp_forward, MTP on vs off, k=1..4 at S=16 and S=2200:tokens_identical=1 logits_identical=1, and every rollback case iscache_and_logits_identical=1(gfx1151, with the #774 RoPE fusion fix).- Verify stays batch-invariant up to eight rows. On RDNA3, k=4 used to fall through to MMQ and diverge.
- Hardcode-bench E4/E6/E7 at a 131K thinking budget: answers byte-identical to MTP off, so scores are unchanged.
Speed (gfx1151, UD-Q4_K_XL)
Decode tok/s at temp 0, fresh server per run, nothing prefix-cached, identical text in every arm; medians of 3 (fixed widths: 3 runs each):
| code 16K | code 64K | ciru counting cell (66K) | |
|---|---|---|---|
| MTP off | 21.8 | 21.0 | 21.2 |
| adaptive (default) | 32.4 (1.49x) | 28.2 (1.34x) | 43.5 (2.05x) |
| fixed k=2 / 3 / 4 | 32.1 / 32.6 / 32.5 | 28.0 / 27.7 / 27.5 |
On the counting cell the adaptive width settles at seven drafts (7.74 tokens per cycle, 445/446 accepted).
- 512-token prompt-type sweep: adaptive 33.3 vs 22.7 tok/s mean (1.46x). The worst row was 1.12x.
- Long hardcode runs (131K thinking, card sampling, temp 1.0): wall 0.88-0.90x of MTP off, accept rate about 0.64.
Prefill
MTP no longer costs prefill. The prompt's MTP K/V is filled inside the trunk's chunk graph (c993ed73, on top of adb04e5c): no separate MTP graphs, no second embedding upload, and no host round-trip of the hidden rows. With #774's prefill stack merged (memory-sized chunks: 7168 rows MTP off, 4864 with MTP at 262K; gfx1151, UD-Q4_K_XL, same session, medians of 2):
| Prompt | MTP off | MTP adaptive | Decode off / adaptive |
|---|---|---|---|
| code 16K | 1,256 tok/s | 1,179 tok/s | 21.8 / 29.7 |
| code 64K | 1,222 tok/s | 1,169 tok/s | 21.0 / 27.2 |
| ciru counting cell (66K) | 1,246 tok/s | 1,188 tok/s | 21.2 / 33.9 (acceptance 1.00) |
MTP output equals MTP off (tokens and logits) at 2048-, 4864- and 7424-row chunks; quality gate HE/GSM/Math 10/10/10 and recall 3/3 with MTP on, greedy 24/24 identical to MTP off.
The MTP K/V bytes are identical to the full MTP layer's, and the smoke checks it along with the trunk cache/logits, the hidden row carried across chunks, a 2048+1 tail and reset/reuse.
Notes
- Gain under sampling. With card thinking sampling (temp 1.0) the decode speedup is 1.13x / 1.18x at 16K / 64K (accept 0.61 / 0.70), against 1.37x / 1.36x at temp 0. Distribution-preserving speculative sampling was tried and gave no gain above noise (accept ~0.69: the MTP head's distribution differs from the trunk's at temp 1.0), so exact-match acceptance stays.
- Stays a draft until #774 lands, since it is stacked on it.