fix(hip): skip the fused-path BF16-only marks when GGML_CUDA_DISABLE_FUSION is set

Merged
GR
5d ago
+5-3

Why

GGML_CUDA_DISABLE_FUSION=1 is the switch we reach for when output looks wrong and we want to rule out a fused kernel. On Strix Halo (gfx1151, RDNA3.5) with a qwen4exp model it currently can't be used: every prefill of 512+ rows aborts in ggml_cuda_compute_forward with

MMB: MUL(...) reads BF16-only tensor ... through the unfused path
MMB: ... writes a BF16/F16-only tensor through the unfused path

Cause

Before each graph runs, the qwen4exp HC marking pass in ggml_backend_cuda_graph_compute flags some hyper-connection intermediates as BF16-only, so that they are stored only as BF16 and never in F32:

  • the HC_COMBINE_NORM xn view. One of the consumers it accepts is the xn*gate MUL, which ggml_cuda_hc_gate_mix fuses and which reads xn through the BF16 cache.
  • the gate-mix dst, which is written as BF16 in place by ggml_cuda_hc_gate_mix.

Both marks assume ggml_cuda_hc_gate_mix runs, but that kernel is only reached inside the !disable_fusion block. With fusion off, the plain MUL and gate-mix ops touch the marked tensors, and the guard in compute_forward aborts, as it should.

The routed-GLU pass in the same block already accounts for this with a fusion_off check. The two passes above were missed.

Change

Hoist the existing fusion_off flag to the top of the block and skip the xn and gate-mix marking passes when it is set. Unmarked tensors take the normal F32 path. The routed down-row F16 mark stays as it is: its only reader is HC_COMBINE_NORM itself, not a fused kernel.

The default path (fusion on) is unchanged. The flags are evaluated once and the marking runs exactly as before.

Testing

  • ggml-cuda.cu compiles for gfx1151 (GGML_USE_HIP)
  • gfx1151 + qwen4exp, GGML_CUDA_DISABLE_FUSION=1, prefill ≥512 rows: aborts before, runs after
  • Default path: output identical before and after

The on-device checks are pending: there's no qwen4exp model on the test box yet. This PR stays in draft until they're done.

🤖 Generated with Claude Code