feat(qwen4exp): MTP speculative decoding (stacked on #774)
PR #774 has merged. This PR now contains the MTP delta against main. Review only the MTP work: https://github.com/dusterbloom/lucebox-hub/compare/c320db9a...d54df4bc
What
Speculative decoding for Qwen3.8-Flash-Next with the model's own MTP block. The single MTP layer is reused to chain up to seven drafts per step, and the trunk verifies them in one batched forward.
- The sidecar (
MTP/mtp-*.ggufnext to the model) is found automatically. Without it, decode runs without MTP. --verify-width 0(default): adaptive width, up to k=7 drafts, from per-depth acceptance and a per-request cycle-cost fit.--verify-width 1: MTP off.--verify-width 2..8: fixed k=1..7.- Drafts are cheap: the draft head covers 64,000 vocabulary rows (control/special tokens plus the lowest ids), picks its token on the GPU and feeds the next draft from device memory; catch-up runs attention/FFN only for the last retained row. Drafts are proposals only, so this never changes output.
- No environment variables.
Exactness
A draft is accepted only if it equals the token the trunk samples, so output is byte-identical to MTP off for the same seed and the same prefill chunk size. Rollback restores the KV/recurrent state bit-exactly. (Automatic chunk sizing is chosen from free memory, and loading the MTP sidecar can change it at some context sizes, e.g. 8192 vs 4096 rows at --max-ctx 81920; pin --chunk when comparing MTP on and off. At the recommended --max-ctx 131072 both use 4096.)
test_qwen4exp_mtp: 4,236 acceptance/rollback cases plus config and adaptive policy.test_qwen4exp_mtp_dispatch: 21 cases.test_feature_gate: widths 0..8 accepted, -1/9 refused.smoke_qwen4exp_forward, MTP on vs off, k=1..4 at S=16 and S=2200:tokens_identical=1 logits_identical=1, and every rollback case iscache_and_logits_identical=1.- At 70K context: rollback/catch-up 8/8 (k=1) and 32/32 (k=7) cases identical (
--chunk 2048); server text identical MTP on vs off on a 69,857-token prompt with--chunk 4096pinned (21.0 vs 25.9 tok/s decode). - MTP output equals MTP off (tokens and logits) at 2048-, 4864- and 7424-row prefill chunks.
- Verify stays batch-invariant up to eight rows. On RDNA3, k=4 used to fall through to MMQ and diverge.
- Quality gate HE/GSM/Math 10/10/10 and recall 3/3 with MTP off and adaptive. Hardcode-bench E4/E6/E7 at a 131K thinking budget: answers byte-identical to MTP off.
Speed (gfx1151, UD-Q4_K_XL)
Decode tok/s at temp 0, fresh server per run, nothing prefix-cached, identical text in every arm; medians of 3:
| code 16K | code 64K | counting cell (66K) | |
|---|---|---|---|
| MTP off | 21.8 | 21.0 | 21.2 |
| adaptive (default) | 32.4 (1.49x) | 28.2 (1.34x) | 43.5 (2.05x) |
| fixed k=2 / 3 / 4 | 32.1 / 32.6 / 32.5 | 28.0 / 27.7 / 27.5 |
On the counting cell the adaptive width settles at seven drafts (7.74 tokens per cycle, 445/446 accepted).
- 512-token prompt-type sweep: adaptive 33.3 vs 22.7 tok/s mean (1.46x). The worst row was 1.12x.
- With card thinking sampling (temp 1.0) the speedup is 1.13x / 1.18x at 16K / 64K (accept 0.61 / 0.70). Distribution-preserving speculative sampling was tried and gave no gain above noise, so exact-match acceptance stays. Long hardcode runs (131K thinking, temp 1.0): wall 0.88-0.90x of MTP off.
Prefill
MTP costs 4-6% of prefill: the prompt's MTP K/V is filled inside the trunk's chunk graph (no separate MTP graphs, no second embedding upload, no host round-trip of the hidden rows). With #774's memory-sized chunks (7168 rows MTP off, 4864 with MTP at 262K; same session, medians of 2):
| Prompt | MTP off | MTP adaptive |
|---|---|---|
| code 16K | 1,256 tok/s | 1,179 tok/s |
| code 64K | 1,222 tok/s | 1,169 tok/s |
| counting cell (66K) | 1,246 tok/s | 1,188 tok/s |
Guide: server/docs/QWEN4EXP.md (MTP section).
2026-10-07 restack validation. PR #774 is merged as c320db9a. The cumulative #810 source at a93d8bef, including the material #808 refactor at d54df4bc and adapted #809/#810, was freshly built for gfx1151 and passed the HIP/model gate (sha256:93210dffb2442f36a841a066f10eb298898a84e389b94c55f2993bcde61e66f1): actual DS4 fused GPU fixture, MTP host/dispatch contracts, Q2_0 differential, MMID, trim/recompute, park/unpark, plain/split/stable forward, all-width MTP rollback, batched and four-slot sequence/HTTP serving, and both model-backed prefix-cache tests. The throughput tables above retain their original measurement attribution.
For #808, this is cumulative stack evidence; its MTP-only source boundary is the compare link above.