feat(qwen4exp): exact 4-slot concurrent serving (stacked on #809)
Stacked on the prefix-cache PR (on MTP; PR #774 is merged). Merge order: #808, #809, this. Review only the concurrency work and its merge onto the stack: https://github.com/dusterbloom/lucebox-hub/compare/f89be47d...4a5a16f5
What
Up to four concurrent sequences for Qwen3.8-Flash-Next on the sequence engine. Each slot has its own KV, recurrent and QSA indexer state. Every slot produces exactly what it would produce alone.
luce_server <model> --max-concurrency 4 --max-ctx 32768
--max-concurrency 1keeps the single-sequence path (MTP, prefix cache). 2..4 enables the sequence engine; more than 4 is refused. No environment variables.- With more than one slot, MTP is switched off (
--draft/--verify-width > 1together with slots are refused) and the prefix cache stays single-slot: the sequence engine replaces the serial loop that owns both. - Prefill chunk: auto-sized for all slots (the planner reserves the other slots' caches), capped at the single-slot target of 4,096 rows. Uncapped it chose 12,800 rows at 4x32K, which stalls the other slots' decode and peaked at 93 GB GTT.
How it stays exact
- Batched decode. A decode step is batched only when every slot is in the exactly-dense attention regime (QSA is exactly dense up to 2,051 tokens) and there are at most four rows. It runs in MMVQ batch-invariant mode, so a four-row batch uses the same kernels and arithmetic as a single row.
- Solo fallback. Any other step, and every prefill, runs each slot through the solo forward. Chunks are interleaved, so the first tokens arrive together.
- The tests follow #774's stable-QSA cache contract: the masked bucket suffix must stay finite and
indexer_k's last row is the decode graph's permanent scratch row.
Tests (gfx1151, UD-Q4_K_XL, strict: bit-exact vs solo, on the full stack)
smoke_qwen4exp_batched M 2050 3072 1: every rowmax_abs_delta=0in the identical and distinct phases; untouched-slot isolation, row permutation and cancel-reset-reuse pass.test_qwen4exp_seq_engine M 32768: N=2/3/4 contract + soak pass, the QSA boundary crossing passes (tokens and indexer state bit-exact), distinct concurrent sequences pass.qwen4exp_concurrency_http.pyagainst default flags: four parallel requests at 2.3K and 16K prompts, 8/8 identical to solo.test_feature_gate,test_server_unit test_qwen4exp,test_qwen4exp_mtp,test_qwen4exp_chunk,test_qwen4exp_prefix_cachepass.
Throughput (gfx1151, UD-Q4_K_XL, --max-ctx 32768, 4,096-row chunks)
| 1 slot | 4 parallel | |
|---|---|---|
| prefill, 2.3K prompts | 1,025 tok/s | 1,096 tok/s aggregate |
| prefill, 16K prompts | 1,179 tok/s | 1,185 tok/s aggregate |
| decode | 24.7-24.9 tok/s | 6.3-6.6 per stream, 24.8 aggregate |
| peak GTT | 89.1 GB |
Above 2,051 tokens of context each slot runs its own QSA forward, so batching adds little to decode there.
Not in this PR
- Batched QSA decode, for real decode throughput above 2K context.
- Per-slot MTP and per-slot prefix snapshots.
- More than four slots.
2026-10-07 stacked validation. The cumulative parent at a93d8bef passed the full gfx1151 HIP/model V7 gate (result sha256:93210dffb2442f36a841a066f10eb298898a84e389b94c55f2993bcde61e66f1). The only follow-up source change restores main's four-line backend-neutral CMake registration for test_ds41_mmid_width_invariance; the fixture source, runtime, kernels, and model code are unchanged. The exact follow-up tree 3bdbeee6 compiled and the restored fixture passed on gfx1151 (result sha256:bcb945b7c3d749ac239269f3f7f553bd707d92a977dbba1747089b16b7c1a906; binary sha256:75883d3798beb1a1f68f6b2a3669c855813a84468395c9548a245fc744dfd73f). The #810 candidate at 4a5a16f5 is byte/mode-identical to that tested tree. The throughput tables above retain their original measurement attribution.
This is the exact cumulative source tree covered by those gates.