feat(qwen4exp): exact 4-slot concurrent serving (stacked on #17)
Draft
DU
7d ago+1893+1893+0123456789012345678901234567890123456789-13-13-01234567890123456789
Stacked on #809 (prefix cache, on #808 MTP, on #774). Merge order: #774, #808, #809, this. Review only the concurrency work and its merge onto the stack: https://github.com/dusterbloom/lucebox-hub/compare/485a8e40...15073cfa
What
Up to four concurrent sequences for Qwen3.8-Flash-Next on the sequence engine. Each slot has its own KV, recurrent and QSA indexer state. Every slot produces exactly what it would produce alone.
luce_server <model> --max-concurrency 4 --max-ctx 32768
--max-concurrency 1keeps the single-sequence path (MTP, prefix cache). 2..4 enables the sequence engine; more than 4 is refused. No environment variables.- With more than one slot, MTP is switched off (
--draft/--verify-width > 1together with slots are refused) and the prefix cache stays single-slot: the sequence engine replaces the serial loop that owns both. - Prefill chunk: auto-sized for all slots (the planner reserves the other slots' caches), capped at the single-slot target of 4,096 rows. Uncapped it chose 12,800 rows at 4x32K, which stalls the other slots' decode and peaked at 93 GB GTT.
How it stays exact
- Batched decode. A decode step is batched only when every slot is in the exactly-dense attention regime (QSA is exactly dense up to 2,051 tokens) and there are at most four rows. It runs in MMVQ batch-invariant mode, so a four-row batch uses the same kernels and arithmetic as a single row.
- Solo fallback. Any other step, and every prefill, runs each slot through the solo forward. Chunks are interleaved, so the first tokens arrive together.
- The tests follow #774's stable-QSA cache contract: the masked bucket suffix must stay finite and
indexer_k's last row is the decode graph's permanent scratch row.
Tests (gfx1151, UD-Q4_K_XL, strict: bit-exact vs solo, on the full stack)
smoke_qwen4exp_batched M 2050 3072 1: every rowmax_abs_delta=0in the identical and distinct phases; untouched-slot isolation, row permutation and cancel-reset-reuse pass.test_qwen4exp_seq_engine M 32768: N=2/3/4 contract + soak pass, the QSA boundary crossing passes (tokens and indexer state bit-exact), distinct concurrent sequences pass.qwen4exp_concurrency_http.pyagainst default flags: four parallel requests at 2.3K and 16K prompts, 8/8 identical to solo.test_feature_gate,test_server_unit test_qwen4exp,test_qwen4exp_mtp,test_qwen4exp_chunk,test_qwen4exp_prefix_cachepass.
Throughput (gfx1151, UD-Q4_K_XL, --max-ctx 32768, 4,096-row chunks)
| 1 slot | 4 parallel | |
|---|---|---|
| prefill, 2.3K prompts | 1,025 tok/s | 1,096 tok/s aggregate |
| prefill, 16K prompts | 1,179 tok/s | 1,185 tok/s aggregate |
| decode | 24.7-24.9 tok/s | 6.3-6.6 per stream, 24.8 aggregate |
| peak GTT | 89.1 GB |
Above 2,051 tokens of context each slot runs its own QSA forward, so batching adds little to decode there.
Not in this PR
- Batched QSA decode, for real decode throughput above 2K context.
- Per-slot MTP and per-slot prefix snapshots.
- More than four slots.