PR #774 - Luce-Org/lucebox

Summary

Qwen3.8-Flash-Next (qwen4exp) as a hand-written target, tuned for the Strix Halo (gfx1151, 128 GB unified memory). The model is complete: 36 Gated DeltaNet and 12 full-attention layers with Qwen Sparse Attention (top-512 blocks of 4 tokens), 4-stream hyper-connections, per-layer n-gram embeddings and a 512-expert MoE (10 active). It is byte-exact against upstream llama.cpp's reference path. Other models are unchanged.

Model support (server/src/qwen4exp/)

Loader, graph, cache and backend. QSA at prefill and decode, with keys pooled over 4-token blocks as in the reference model.
Chat template: the GGUF's official Jinja template, thinking by default (xhigh effort); reasoning_effort low / medium / xhigh pass through. Multi-turn replays reasoning_content and honours chat_template_kwargs.preserve_thinking.
Thinking-budget force-close (Qwen's own budget text). Cards carry sampling_no_thinking (Qwen's instruct set) for requests with thinking off.

Kernels (ggml-cuda/, HIP)

QSA selected attention; MMB bf16 and Q8_0 -> F16 WMMA GEMMs with an F16 activation dataflow; HC / GDN / PLE fusion; M-RoPE written straight into the flash-attention layout; packed get_rows; head-256 MMA flash attention; MMQ for IQ3_S / IQ4_XS / Q4_K / Q5_K / Q6_K / Q2_0.

Serving

No environment variables: on gfx1151 the backend selects its measured kernel profile in code, scoped to qwen4exp, so other models keep their dispatch. The upstream reference path used by the differential is a test/script argument.
Without --chunk, prefill chunks are sized to the free memory (7424 rows for UD at 262K context, one slot). Prompt attention accumulates in F32: on UD an 18K prompt prefilled in 2048- or 7424-row chunks gives logits bitwise equal to one pass.

Models

Base model: Qwen/Qwen3.8-Flash-Next. GGUFs verified on this branch:

UD-Q4_K_XL: unsloth/Qwen3.8-Flash-Next-GGUF (UD-Q4_K_XL/)
IQ4_NL: bartowski/Qwen3.8-Flash-Next-GGUF
GSQ-RCO IQ3_XXS: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF (IQ3_XXS/)

Results (Strix Halo gfx1151, default flags, fresh server per run)

	prefill	decode	GTT
UD-Q4_K_XL, 16K prompt	1,200-1,240 tok/s	22.4 tok/s	77 GB
UD-Q4_K_XL, 64K prompt	1,200-1,250 tok/s	20.9 tok/s
UD-Q4_K_XL, short prompt	-	24.1 tok/s
IQ4_NL, 16K prompt	950-1,010 tok/s	28.5 tok/s (30.6 short)	73 GB
GSQ-RCO IQ3_XXS, 16K prompt	~475 tok/s	25-26 tok/s	47 GB

UD prefill rows: memory-sized chunks at 262K context; the other quants were measured with 2048-row chunks.

Quality (lucebox gates, prompts above the 512-token MMB floor with a 1,500-token preamble): HE / GSM / Math 10 / 10 / 10 and 16K planted-fact recall 3/3 on all three quants.

Validation

Upstream differential (reference path): no onset and 0 expert-ID mismatches across all 48 layers on IQ4_NL and UD-Q4_K_XL; no onset on GSQ-RCO IQ3_XXS.
Split-prefill KLs reproduce exactly (0.082874 / 0.118543 / 0.652715 / 0.734763); T=1 decode and prefill snapshots are byte-identical across 3 fresh processes.
test_qwen4exp_qsa_ids (89 cases, GPU and CPU), test_qwen4exp_indexer_score, test_rope_tail (including the M-RoPE -> CONT position alias), test_backend_plan, test_server_unit.
CI: Linux build, Windows MSVC + CUDA, R9700, Strix Halo and DGX GB10 green.

Known limits

Kernels, defaults and quality gates are tuned and measured on gfx1151 only.
One sequence at a time: --max-concurrency > 1 is refused. Exact 4-slot serving (dusterbloom/lucebox-hub#16) and MTP speculative decoding (dusterbloom/lucebox-hub#15) follow as separate PRs on top of this one.
Hard reasoning tasks need more thinking than the card default (32K): on hardcode-bench E4 (UD) the model scores 0/13 at 32K and 12/13 at 131K (--max-ctx 151552 --think-max-tokens 131072).

Guide: server/docs/QWEN4EXP.md.