feat(qwen4exp): split Qwen3.8-Flash-Next across an R9700 and a Strix Halo

Merged
DA
7d ago
+3049-916

Runs Qwen3.8-Flash-Next with the routed experts on a second GPU. On an R9700 + Strix Halo box, the R9700 holds the dense weights, the KV cache, the MTP drafter and the most-used experts, and the Strix Halo holds the rest of the experts. Both GPUs run under one scheduler.

Results

UD-Q4_K_XL, adaptive MTP, greedy, 131K context, 8 GiB of hot experts. Same harness for both columns.

Strix Halo alone (main)R9700 + Strix Halo
Decode, short reply40.5 tok/s75.5 tok/s (1.86x)
Decode, code reply41.4 tok/s63.1 tok/s
Decode after a 16K prompt24-28 tok/s44-45 tok/s
Prefill, 16K prompt1,220-1,320 tok/s1,820 tok/s
Prefill, 55K prompt1,310 tok/s1,760 tok/s
Same 16K prompt sent again (prefix cache)2.7 s instead of 10.6 s

Without --expert-placement, the split decodes a short reply at 69 tok/s.

How to run

luce_server Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
  --profile qwen-next-r9700-strix --expert-placement routes.csv
  • --expert-device hip:1: the existing DeepSeek V4 flag now also places Flash-Next's routed experts. It must differ from the target device.
  • --expert-placement PATH: one flag for every expert tier. DeepSeek V4 reads its owner-map JSON; Flash-Next with --expert-device reads a routing-stats CSV and copies the most-routed experts to the target, where decode and verify steps run them. --ds4-expert-placement stays as an alias. The CSV is an operator input and is not part of this PR.
  • LUCE_EXPERT_BUDGET_MB (default 8192 for Flash-Next): the same budget DeepSeek V4 and Laguna use, read through one helper.
  • --profile qwen-next-r9700-strix: target hip:0, experts on hip:1, --max-ctx 131072.
  • Build both architectures (gfx1151;gfx1201). -DGGML_HIP_GRAPHS=ON adds about 3%.

Split mode serves one sequence. --max-concurrency above 1 is refused together with --expert-device until the multi-slot graph gets split placement (follow-up PR).

Commits

  1. feat(qwen4exp): split mode runs the routed experts on a second GPU. Prompt chunks pipeline over 2 to 4 streams, so one GPU works on one stream while the other works on the next. QSA runs on RDNA4.
  2. perf(cuda): qwen4exp's generic fusions on every device of the graph.
  3. perf(qwen4exp): MTP verify and draft steps replay retained graphs.
  4. perf(qwen4exp): linear-attention steps run fewer kernels.
  5. perf(qwen4exp): verify attends every sparse-attention row in one pass (block-radix top-k on gfx1201 too).
  6. fix(qwen4exp): the adaptive verify width keeps its cost per context range and its acceptance across requests.
  7. feat(server): the flags above replace the old environment variables.
  8. fix(qwen4exp): split mode with prefix snapshots. A restore also clears the retained verify and draft graphs, and split mode keeps an 8192-row prompt chunk ahead of the snapshot allowance.
  9. fix(qwen4exp): single-token steps run their hot experts on the target too, so greedy output does not depend on the verify width.
  10. fix(server): the prefix cache's automatic limit stays within the backend's snapshot allowance. Before this fix, a full allowance refused every new capture and evicted nothing.
  11. docs(qwen4exp): split mode section.

Cleanup on top, so nothing runs in two parallel copies:

  1. refactor(spec): Flash-Next's MTP verify width runs on the shared AdaptiveSpecWidth. The Flash-Next controller classes are gone. The shared class gains opt-in pieces (shape seed, per-depth acceptance, cost per context range, carried acceptance). DeepSeek V4's defaults are unchanged: a replay test checks 800 of its width decisions against the previous controller.
  2. refactor(server): one --expert-placement and one budget for every expert tier, replacing the Flash-Next-only flags.
  3. refactor(qwen4exp): one router and shared-expert builder for both MoE paths.
  4. refactor(qwen4exp): single-stream and pipelined layers share one per-layer builder.
  5. refactor(qwen4exp): one place decides which retained graphs a position jump drops.
  6. refactor(qwen4exp): split chunk limits live in the chunk planner, plus a checked table allocation and named constants.
  7. fix(qwen4exp): unpark brings the hot experts back and sizes the chunk and snapshot allowance for the memory free at that moment. Init and unpark share one load_target(); park, shutdown and a failed unpark share release_target().
  8. refactor(qwen4exp): one expert-device check (in the server), a named hot-expert budget, and one arena for the small derived tensors.
  9. docs(qwen4exp): UD-Q4_K_XL downloads from the Lucebox repo (a byte-identical copy of the Unsloth files, with the MTP drafter).
  10. refactor(qwen4exp): forward_impl builds from named pieces (918 to 335 lines): a Qwen4ExpLayerBuilder shared by single-stream layers and pipeline streams, the pipelined stage loop, and the stable-graph upload and replay as their own functions. Code motion only; greedy text identical.
  11. docs(readme): Qwen 3.8 Flash-Next in the supported-models table (Strix Halo, and R9700 + Strix Halo) and the heterogeneous results.

Validation

  • Greedy output: identical with adaptive MTP, a fixed --verify-width 4 and MTP off (8/8 prompts). On one GPU the branch matches main (8/8). The cleanup commits keep greedy output identical to the commit before them, in split and on one GPU (8/8 each).
  • Units: test_qwen4exp_mtp, _mtp_dispatch, _chunk, _prefix_cache, test_server_unit (with the new split-placement plan test). test_qwen4exp_indexer_score, _qsa_ids and _qsa_attn pass on both gfx1201 and gfx1151.
  • Split smoke (MTP, 2600 tokens): tokens and logits identical, rollback 16/16.
  • Park/unpark in split mode: the hot experts are resident again after unpark, and greedy text is identical across it (model lifecycle test).
  • Quality: bench checks 6/6.

🤖 Generated with Claude Code