feat(ds4): add concurrent serving on Strix Halo and R9700 + Strix
Summary
Adds paged concurrent serving for DeepSeek V4 on Strix Halo and on an R9700 target with Strix Halo expert execution. Multiple requests share each model weight pass while retaining separate attention history, compressor state, sampling state, and request lifecycles.
The PR also reduces graph construction, kernel launches, and memory traffic in this serving path, and tunes the existing HIP kernels for the small batches produced by concurrent decode.
Concurrent serving
- Model-specific sequence engine.
DeepSeek4SeqEngineimplements the shared scheduler interface. Each step advances live decoders and selected prefills, with up to six live sequences and chronological prompt-row packing on the monolithic path. The existing HTTP scheduler handles admission, cancellation, streaming, and slow clients. - Paged model state.
DeepSeek4PagedCachekeeps a 128-token raw attention ring per slot, compressed history in physical pages, and separate compressor state for each sequence. Block tables map logical history to physical rows, so requests can reuse released pages without sharing state. - Shared slot ownership.
SeqSlotManagercentralizes the host-side lifecycle used by Qwen and DeepSeek: prompt reservations, decode headroom, page allocation, sampling history, seeded RNGs, and retirement. It preserves seed zero and releases active pool handles on destruction. - Gathered attention graphs. Each lane gathers its own chronological history and runs the existing attention computation. Dense projections, hyper-connection operations, and expert execution share the batch. Each token updates only the owning sequence's state.
- Memory accounting. Paged serving allocates its own cache instead of also retaining the unused contiguous cache. Default capacity rounds each sequence's context to whole pages, and hybrid placement budgets for the tensors actually allocated.
Optimizations
- Reuse graphs across decode steps. Heterogeneous decode uses masked history buckets to keep graph shapes stable. Prefill and monolithic serving retain exact history shapes. Exact graphs reuse their structure when history sizes and compressed-row emission match; positions, gather indices, and write destinations remain fresh device inputs. Short-lived monolithic graphs execute without HIP capture overhead.
- Reduce expert-route overhead. The six-route split uses a native two-route tail, non-owner routes use negative sentinels, and route slices avoid per-token copy/concatenation chains. The qualified dual-GPU recipe enables existing grouped expert execution and split-copy batching.
- Preserve scheduler resources safely. Graph rebuilds retain compatible scheduler staging allocations, retire native graphs before replacing their metadata, and track copy-destination readiness per backend. Property-check bypass requires a matching nonzero graph generation. Successful graph reuse no longer logs every hit.
- Tune small HIP batches. On gfx1151, narrow F16 projections stay on MMVF through eight columns, paged dense ROCmFP4 projections use the weight-reuse MMVQ kernel from two through sixteen columns (each column keeps the single-column arithmetic; two and three columns previously took the generic per-column kernel), and the packed FP3 expert loader is enabled by default. Mixed FP2/FP3 expert kernels reduce load overhead; wider FP2 gate/up execution folds SwiGLU into its second projection pass. Existing explicit dispatch overrides remain available.
- Batch compressor projections. Monolithic paged serving prepares compressor value and gate projections once at the gathered width for ROCmFP4 weights. Per-lane column views feed the existing compressor update, preserving chronological state writes, pooling, and rotation.
- Fuse tail RoPE.
GGML_ROPE_TYPE_TAILrotates the final rotary dimensions and copies the remaining head dimensions in one operation. CPU and CUDA/HIP implementations replace DeepSeek's split, copy, rotate, and concatenate sequence while retaining its arithmetic.
Supported scope
Monolithic paged serving supports Strix Halo. Heterogeneous serving requires an R9700 primary and an in-process Strix Halo expert backend with static expert ownership. The runtime supports up to six lanes; the documented grouped-expert recipe is qualified through four clients with ROCmFP2 gate/up and ROCmFP3 down weights.
- Pack prompt rows. On the monolithic exact path concurrent prompts advance up to four chronological rows per sequence per scheduler step, with sixteen rows total. A lone prompt with no active decoder can use all sixteen rows. Each row is its own gathered lane that reads the rows before it in graph order, so attention, the raw ring and the compressor state stay exact; dense projections run once for all rows on the sixteen-column kernel, narrow F16 mixes in eight-column parts, and routed experts in parts of up to eight rows (one-row parts for a single live sequence). Pure decode rounds keep their graph. The legacy CUDA buffer pool keeps its free buffers in a size-ordered map with 8192 slots, since a packed step memoizes more than the old 1024 activations.
Paged serving requires exact prefill and autoregressive decode. Unsupported combinations fail initialization, including drafts/DSpark, fused decode, approximate prefill, explicit layer splits, remote shards, mutable expert caching, and prefix-cache parking. Heterogeneous serving still advances one prompt token per lane per step.
MoE arithmetic can still vary with the live lane count, so arbitrary batch compositions are not guaranteed to produce identical tokens.
Strix Halo monolithic results on 2026-09-06 (fixed-C A/B/A against 644c804b, exact output at every width)
Two- and three-column ROCmFP4 weight reuse (3dd7eef7): C2 output-window throughput +4.09% (drift 0.10%), C3 +7.92% (drift -0.31%), p95 TTFT -2.84% and -6.59%.
Packed prompt rows with the sixteen-column kernel and eight-row expert parts (cd33b50d, 74a6b203, bc80cbfc), measured together:
| C | Output-window tok/s before | after | Change | Drift | p95 TTFT change |
|---|---|---|---|---|---|
| 1 | 23.48 | 23.51 | +0.11% | +0.09% | -26.82% |
| 2 | 33.17 | 34.43 | +3.81% | -0.89% | -32.23% |
| 3 | 37.65 | 40.56 | +7.75% | +0.08% | -22.47% |
| 4 | 45.69 | 45.32 | -0.81% | -0.19% | -9.48% |
On four distinct prompts per wave (mixed-36.jsonl, two waves, 128 output tokens, baseline bracketed) the packed runtime raises output-window throughput 17.81 to 22.95 tok/s (+28.9%, 0.0% drift), aggregate 15.31 to 18.73, and cuts median TTFT 11.18 to 8.50 s and the 2k-prompt maximum 29.9 to 20.0 s, because decoders spend fewer rounds beside a one-token-per-round prefill. The C4 decode change is a uniform ~1% slowdown of every kernel after a packed prefill with identical launch counts, not a code-path difference. The replicated-prompt fixture understates the C4 TTFT gain: the baseline's expert kernel runs 17-46% faster per call on four identical tokens than on four distinct ones. Records: Luce-Org/luce_box PR 95.
Strix Halo monolithic follow-up on 2026-09-07
The gfx1151 paged default now sets LUCE_MMVQ_MAX_NCOLS=16, restoring the dense ROCmFP4 route required by packed prefill while honoring an explicit environment setting. The gathered graph's capture setting does not override this ceiling.
Lone-prompt packing (bcf5263c) was measured against the corrected four-row baseline (fbce7c5b) in fresh fixed-C A/B/A brackets:
| C | p95 TTFT change | Output-window throughput change | Saved output matches |
|---|---|---|---|
| 1 | -5.90% | -0.54% | 3/3 |
| 2 | -0.19% | -0.09% | 6/6 |
| 3 | +0.04% | +0.06% | 9/9 |
| 4 | -0.41% | 0.00% | 12/12 |
All 30 candidate responses and both baseline arms match saved 039dc950 outputs, including content and reasoning hashes, token counts, and finish reasons. Each concurrency uses the same three replicated-prompt waves. C1's 5.90% TTFT reduction clears twice its 0.82% baseline TTFT drift, with a measured 0.54% output-throughput cost. C2–C4 remain effectively unchanged.
Using spare rows for concurrent prompts was also tested and excluded: uneven C3 progress reduced output throughput by 2.32% and increased TTFT by 1.95%. The final policy widens only a lone prompt, using existing slot state and no new flag. Projection partition widths and single-sequence expert arithmetic stay unchanged.
Focused tests cover second admission, active decoding, retirement, and heterogeneous limits. GPU arithmetic tests cover every ROCmFP4 width from two through sixteen with zero bit mismatches against independent single-column execution.
Gathered attention consolidation
Commit 3b399945 consolidates gathered attention data movement and inverse tail RoPE, uses exact sink-column softmax for histories up to 2048 rows, and reuses the two-scatter compressor flush. Repeated-slot prompt rows preserve chronological cache reads; hybrid context scratch is explicitly assigned to the main backend.
The final unprofiled Strix C4 A/B/A bracket measured +3.54% output-window throughput against -0.38% baseline drift, with 1.35% lower p95 TTFT. Matched LuceGraph captures show 15.82% fewer decode launches and 17.97% fewer full packed-step launches. Profiled timing is diagnostic.
Initial C1-C4 monolithic checks matched all 30 saved outputs. The guarded revision matched all 12 final C4 outputs; the final hybrid ownership repair matched all 30 C1-C4 dual-GPU outputs. The monolithic bracket and profiles precede the three host-only hybrid placement lines. Focused sink tests pass on CPU, gfx1151 and gfx1201; tail RoPE and DS4 unit tests pass. No new runtime switches. Arbitrary batch invariance and long-history performance are not claimed.
Compressor projection batching on 2026-09-07
Commit a085cc22 adds 33 lines and removes 4 across two graph-builder files. It reuses the existing projection helper and prepared-lane bundle, with no new kernel, runtime flag, or persistent state. Qualification is on gfx1151 monolithic paged serving against the preceding PR head, ebf848f3.
| C | A1 output tok/s | B output tok/s | A2 output tok/s | B vs bracket | A2 vs A1 drift | p95 TTFT change | Exact output |
|---|---|---|---|---|---|---|---|
| 1 | 23.63 | 23.69 | 23.60 | +0.33% | -0.13% | -1.88% | yes |
| 2 | 34.78 | 35.36 | 34.85 | +1.57% | +0.20% | -2.97% | yes |
| 3 | 42.00 | 42.96 | 42.05 | +2.23% | +0.12% | -3.07% | yes |
| 4 | 47.71 | 48.68 | 47.50 | +2.25% | -0.45% | -2.06% | yes |
Each concurrency uses three replicated-prompt waves at 96 generated tokens per request in baseline/candidate/baseline order. All 90 requests meet the fixed-token workload check, and candidate output bytes match both baselines. C1 is essentially unchanged; C2-C4 throughput gains clear twice their measured drift.
The table reports canonical aggregate output-window throughput. The preregistered median-per-wave C4 gate separately passes: +2.23% throughput against -0.59% baseline drift, exceeding the original 2% threshold and twice drift.
A separate C4 bracket uses twelve distinct prompts per arm with natural EOS and a 128-token cap. All 36 responses match across the bracket. Output-window throughput improves 1.90% (drift -0.21%), server-native prefill throughput improves 2.19% (drift -0.27%), and p95 TTFT falls 1.93% (drift +0.36%). These distinct-prompt improvements clear twice their respective drift. Fixed-C4 prefill changes are closer to noise.
The reduced four-layer real-weight proof compares 318 checkpoints and 2.16 GB of logits and cache/state per run, covering C1-C5 decode, packed widths 4/8/12/16, compression ratios 4/128, and raw-ring wrap. Matched C4 profiles remove 20 decode and 92 packed launches per four-layer step. These are reduced-model launch counts; no full-model traffic-counter claim is made. Full-model qualification covers C1-C4 on gfx1151, not hybrid or other-device performance.
Validation
Focused tests cover page addressing, cache planning, slot ownership, scheduler contracts, feature gates, graph generations, expert kernels, width-four/five dispatch, and tail RoPE. Real-model checks on both supported GPU configurations cover mixed-length concurrent requests, page boundaries, cancellation, slot reuse, and repeatable seeded sampling.
The sink-column review fixes cover rows beyond shared-memory capacity and
reject unsupported sink-column mode on Vulkan, Metal, OpenCL, SYCL, and
WebGPU so the scheduler can select a supporting backend. The expanded
CPU/HIP regression passes on gfx1151 and gfx1201 with zero bit mismatches
against scale-concat-softmax, including shared-memory boundary widths and
distinct row sinks. Other backend capability guards were reviewed but could
not be runtime-tested with the available toolchains.
Contributor credits and history
The two consolidated Graffioh-authored commits above origin/main (b1c88230) remain unchanged: c00b8c10 (perf(ggml)) for backend operations, kernels, and their tests; ebf848f3 (perf(ds4)) for paged concurrent serving and gathered execution. The qualified compressor projection follow-up, a085cc22, is appended without rewriting either commit.
The earlier history cleanup preserved the validated source tree. Earlier hashes in the measurements above identify the historical revisions used for those experiments.
The original contributions by cheese-cakee (42ae0293, 88f09687:
graph-generation safety and regression coverage) and poruru-code
(b4de7455: portable Windows/HIP environment handling) are already in main
and remain intact in this branch's ancestry with their original authorship.
Existing co-author credits are retained in the consolidated commits.