perf(ggml-hip): SWAR __vsub4 and __vcmpne4 for the IQ2/IQ3 decoders
Summary
Backport of ggml-org/llama.cpp#27962. On HIP, the fallbacks for the CUDA byte-SIMD intrinsics __vcmpne4 and __vsub4 compiled to a per-byte loop and a saturating subtract. This replaces them with SWAR bit arithmetic in ggml-cuda/vendors/hip.h.
The only callers are the IQ2_XXS/XS/S and IQ3_XXS/S decoders in the MMVQ and MMQ paths (20 call sites). All of them apply signs as (g ^ s) - s, with s in {0x00, 0xff} per byte and grid bytes of at most 62, so the wrapping subtract never differs from the saturating one. The decoded weights are bit-identical.
This matters for the published Qwen3.8 file: UD-IQ4_XS keeps 2.4 GB of its tensors in IQ3_S/IQ3_XXS/IQ2_S/IQ2_XS. At the 16-row verify shape those ran at 117-240 GB/s, vs 444 GB/s for IQ4_XS (rocprofv3 on an R9700).
Results
R9700 (gfx1201), Qwen3.8-27B UD-IQ4_XS + DFlash2 q8_0, published profile (--draft-block-size 16 --max-ctx 131072 --cache-type-k q8_0 --cache-type-v q8_0). Both sides include #765:
| before | this PR | |
|---|---|---|
| output sha256, 11 prompts (5 code, 4 chat, 26.7K-token context, warm-up) | identical on all 11 | |
| decode, per-request wall (same step counts) | 2.623 / 4.838 s | 2.527 / 4.667 s (-3.5 to -3.7%) |
| 26.7K-token prefill | 31.8 s | 30.9 s (-2.8%) |
Upstream measured +21.6% tg128 on a Qwen3.8-27B UD-IQ3_S file on gfx1201.
Test plan
- Byte-identical outputs on the prompt set above (R9700)
- Every
__vsub4caller audited for saturation (none reachable) - CI
🤖 Generated with Claude Code