feat(qwen4exp): exact 4-slot concurrent serving (stacked on #17)

Draft
DU
7d ago
+1893-13

Stacked on #809 (prefix cache, on #808 MTP, on #774). Merge order: #774, #808, #809, this. Review only the concurrency work and its merge onto the stack: https://github.com/dusterbloom/lucebox-hub/compare/485a8e40...15073cfa

What

Up to four concurrent sequences for Qwen3.8-Flash-Next on the sequence engine. Each slot has its own KV, recurrent and QSA indexer state. Every slot produces exactly what it would produce alone.

luce_server <model> --max-concurrency 4 --max-ctx 32768
  • --max-concurrency 1 keeps the single-sequence path (MTP, prefix cache). 2..4 enables the sequence engine; more than 4 is refused. No environment variables.
  • With more than one slot, MTP is switched off (--draft / --verify-width > 1 together with slots are refused) and the prefix cache stays single-slot: the sequence engine replaces the serial loop that owns both.
  • Prefill chunk: auto-sized for all slots (the planner reserves the other slots' caches), capped at the single-slot target of 4,096 rows. Uncapped it chose 12,800 rows at 4x32K, which stalls the other slots' decode and peaked at 93 GB GTT.

How it stays exact

  • Batched decode. A decode step is batched only when every slot is in the exactly-dense attention regime (QSA is exactly dense up to 2,051 tokens) and there are at most four rows. It runs in MMVQ batch-invariant mode, so a four-row batch uses the same kernels and arithmetic as a single row.
  • Solo fallback. Any other step, and every prefill, runs each slot through the solo forward. Chunks are interleaved, so the first tokens arrive together.
  • The tests follow #774's stable-QSA cache contract: the masked bucket suffix must stay finite and indexer_k's last row is the decode graph's permanent scratch row.

Tests (gfx1151, UD-Q4_K_XL, strict: bit-exact vs solo, on the full stack)

  • smoke_qwen4exp_batched M 2050 3072 1: every row max_abs_delta=0 in the identical and distinct phases; untouched-slot isolation, row permutation and cancel-reset-reuse pass.
  • test_qwen4exp_seq_engine M 32768: N=2/3/4 contract + soak pass, the QSA boundary crossing passes (tokens and indexer state bit-exact), distinct concurrent sequences pass.
  • qwen4exp_concurrency_http.py against default flags: four parallel requests at 2.3K and 16K prompts, 8/8 identical to solo.
  • test_feature_gate, test_server_unit test_qwen4exp, test_qwen4exp_mtp, test_qwen4exp_chunk, test_qwen4exp_prefix_cache pass.

Throughput (gfx1151, UD-Q4_K_XL, --max-ctx 32768, 4,096-row chunks)

1 slot4 parallel
prefill, 2.3K prompts1,025 tok/s1,096 tok/s aggregate
prefill, 16K prompts1,179 tok/s1,185 tok/s aggregate
decode24.7-24.9 tok/s6.3-6.6 per stream, 24.8 aggregate
peak GTT89.1 GB

Above 2,051 tokens of context each slot runs its own QSA forward, so batching adds little to decode there.

Not in this PR

  • Batched QSA decode, for real decode throughput above 2K context.
  • Per-slot MTP and per-slot prefix snapshots.
  • More than four slots.