@FIR-2152 - Enable generic mixed-precision MAT_MUL offload by packing…

Merged
AK
3mo ago
+276-73

##########
TSISIM RESULT
root@tsisim:/usr/bin/tsi/bin# ./run_llama_cli.sh
MODEL: tinyllama-vo-5m-para.gguf
INFO: updated /usr/bin/tsi/bin/tsi-ggml/tsavorite-model-deployment.yaml with txe_count:20, multi_thread_enable:true; preserved advanced_matmul_shape_offload:true, triton_matmul_small_n_transpose_opt:true

TSI deploy yaml=tsavorite-model-deployment.yaml txe_count=20 multi_thread_enable=1 user_dram_size_gb=16 user_dram_size_bytes=17179869184 advanced_matmul_shape_offload=1 triton_matmul_small_n_transpose_opt=1
[2026-08-05 20:44:57.832] [info] </proj/work/srv-ghr/ral/FlexHAL/src/hal.c:170> hal_lib_init: topology from tsi-soc-hal: ral_tag=SKYLP_G0741 num_chiplets=10 chiplet_stride=0x1c000000 ral_csr_base=0x20000000
[2026-08-05 20:44:57.837] [info] </proj/work/srv-ghr/ral/FlexHAL/src/hal.c:263> hal_lib_init: complete — 10 chiplets, ral_tag=SKYLP_G0741
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:823> num_vm_segments: 2
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:826> VM segment 0: start_addr = 0xffb055105000, size = 17079185408
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:833> VMA: map_addr = 0xffb055105000, map_size = 17079185408
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:835> VMA: dev_buf = 0xffb5682f03b8
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:836> VMA: dev_buf_num = 1
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:837> VMA: dev_buf_offset = 100683776
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:826> VM segment 1: start_addr = 0xffb04f100000, size = 100683776
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:833> VMA: map_addr = 0xffb04f100000, map_size = 100683776
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:835> VMA: dev_buf = 0xffb5682f0490
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:836> VMA: dev_buf_num = 2
[2026-08-05 20:44:57.839] [info] </proj/work/srv-ghr/git-runner/sdk-ci-arm64-actions-runner/_work/sdk/sdk/runtime/third-party/common-runtime/src/cma/cma.c:837> VMA: dev_buf_offset = 0
was Tim. He loved

llama_perf_sampler_print: sampling time = 9.32 ms / 11 runs ( 0.85 ms per token, 1180.38 tokens per second)
llama_perf_context_print: load time = 2848.88 ms
llama_perf_context_print: prompt eval time = 2615.44 ms / 6 tokens ( 435.91 ms per token, 2.29 tokens per second)
llama_perf_context_print: eval time = 8887.19 ms / 4 runs ( 2221.80 ms per token, 0.45 tokens per second)
llama_perf_context_print: total time = 11749.55 ms / 10 tokens

=== GGML Perf Summary ===
Op Target Runs TSI_KERNEL-RUN Total us Avg us
ADD OPU 80 150 13161 164.51
MUL OPU 85 160 14798 174.09
RMS_NORM OPU 85 85 9229 108.58
MUL_MAT CPU 696 0 116249 167.02
MUL_MAT OPU 165 1700 9118989 55266.60
CONT OPU 80 0 2510 31.38
RESHAPE CPU 338 0 246 0.73
VIEW CPU 346 0 39 0.11
VIEW OPU 80 0 104 1.30
PERMUTE CPU 218 0 66 0.30
PERMUTE OPU 80 0 55 0.69
TRANSPOSE CPU 119 0 31 0.26
GET_ROWS OPU 15 0 134 8.93
SET_ROWS CPU 254 0 1082 4.26
SOFT_MAX OPU 40 0 13510 337.75
ROPE CPU 214 0 2235 10.44
GLU OPU 40 75 554598 13864.95

OPU Profiling Results:
Profiler disabled

root@tsisim:/usr/bin/tsi/bin#
root@tsisim:/usr/bin/tsi/bin#
root@tsisim:/usr/bin/tsi/bin#
root@tsisim:/usr/bin/tsi/bin# /tsi/tsi-sw/check_tsictl.sh
Index 0: execute : 8874
execute_debug : 0
Index 1: execute : 2989
execute_debug : 0
Index 2: execute : 2989
execute_debug : 0
Index 3: execute : 2989
execute_debug : 0
Index 4: execute : 2909
execute_debug : 0
Index 5: execute : 2784
execute_debug : 0
Index 6: execute : 2384
execute_debug : 0
Index 7: execute : 2384
execute_debug : 0
Index 8: execute : 2384
execute_debug : 0
Index 9: execute : 2384
execute_debug : 0
Index 10: execute : 2384
execute_debug : 0
Index 11: execute : 2384
execute_debug : 0
Index 12: execute : 2384
execute_debug : 0
Index 13: execute : 2384
execute_debug : 0
Index 14: execute : 2384
execute_debug : 0
Index 15: execute : 2384
execute_debug : 0
Index 16: execute : 2304
execute_debug : 0
Index 17: execute : 2304
execute_debug : 0
Index 18: execute : 2304
execute_debug : 0
Index 19: execute : 1162
execute_debug : 0
root@tsisim:/usr/bin/tsi/bin#

#########
POSIX RESULT
[100%] Linking CXX executable ../../bin/llama-cli
[100%] Built target llama-cli
INFO: fixing GLIBC compatibility for TSI binaries (build-posix)
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$ build-posix/bin/llama-cli -p "My cat's name is" -m /proj/rel/sw/ggml/models/tinyllama-vo-5m-para.gguf --device tsavorite -c 12288 --temp 0.0 --n-predict 1 --repeat-penalty 1.5 -b 1024 --top-k 50 --top-p 0.9 --repeat-last-n 5 --no-warmup --no-conversation

TSI deploy yaml=tsavorite-model-deployment.yaml txe_count=1 multi_thread_enable=1 user_dram_size_gb=8 user_dram_size_bytes=8589934592 advanced_matmul_shape_offload=1 triton_matmul_small_n_transpose_opt=1
My cat's name is Tim

llama_perf_sampler_print: sampling time = 2.22 ms / 8 runs ( 0.28 ms per token, 3598.74 tokens per second)
llama_perf_context_print: load time = 1102.06 ms
llama_perf_context_print: prompt eval time = 670.98 ms / 7 tokens ( 95.85 ms per token, 10.43 tokens per second)
llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_perf_context_print: total time = 1104.63 ms / 8 tokens

=== GGML Perf Summary ===
Op Target Runs TSI_KERNEL-RUN Total us Avg us
ADD OPU 16 100 25228 1576.75
MUL OPU 17 107 91481 5381.24
RMS_NORM OPU 17 17 13016 765.65
MUL_MAT CPU 123 0 32512 264.33
MUL_MAT OPU 33 33 439457 13316.88
CONT OPU 16 0 1330 83.12
RESHAPE CPU 95 0 27 0.28
VIEW CPU 83 0 11 0.13
VIEW OPU 16 0 5 0.31
PERMUTE CPU 62 0 18 0.29
PERMUTE OPU 16 0 2 0.12
TRANSPOSE CPU 27 0 6 0.22
GET_ROWS OPU 3 0 840 280.00
SET_ROWS CPU 60 0 76 1.27
SOFT_MAX OPU 8 0 7627 953.38
ROPE CPU 63 0 325 5.16
GLU OPU 8 50 37294 4661.75

OPU Profiling Results:
Profiler disabled

[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$ build-posix/bin/llama-cli -p "My cat's name is" -m /proj/rel/sw/ggml/models/Tiny-Llama-v0.3-FP32-1.1B-F32.gguf --device tsavorite -c 12288 --temp 0.0 --n-predict 1 --repeat-penalty 1.5 -b 1024 --top-k 50 --top-p 0.9 --repeat-last-n 5 --no-warmup --no-conversation

TSI deploy yaml=tsavorite-model-deployment.yaml txe_count=1 multi_thread_enable=1 user_dram_size_gb=8 user_dram_size_bytes=8589934592 advanced_matmul_shape_offload=1 triton_matmul_small_n_transpose_opt=1
My cat's name is L

llama_perf_sampler_print: sampling time = 2.08 ms / 8 runs ( 0.26 ms per token, 3853.56 tokens per second)
llama_perf_context_print: load time = 121830.23 ms
llama_perf_context_print: prompt eval time = 107706.40 ms / 7 tokens (15386.63 ms per token, 0.06 tokens per second)
llama_perf_context_print: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_perf_context_print: total time = 121834.95 ms / 8 tokens

=== GGML Perf Summary ===
Op Target Runs TSI_KERNEL-RUN Total us Avg us
ADD OPU 44 296 103829 2359.75
MUL OPU 45 303 213007 4733.49
RMS_NORM OPU 45 45 114679 2548.42
MUL_MAT CPU 403 0 1177178 2921.04
MUL_MAT OPU 89 89 102969597 1156961.76
CONT OPU 44 0 7373 167.57
RESHAPE CPU 237 0 217 0.92
VIEW CPU 240 0 22 0.09
VIEW OPU 44 0 12 0.27
PERMUTE CPU 157 0 77 0.49
PERMUTE OPU 44 0 8 0.18
TRANSPOSE CPU 87 0 23 0.26
GET_ROWS OPU 3 0 3204 1068.00
SET_ROWS CPU 175 0 345 1.97
SOFT_MAX OPU 22 0 41071 1866.86
ROPE CPU 174 0 7221 41.50
GLU OPU 22 148 329728 14987.64

OPU Profiling Results:
Profiler disabled

[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$
[akapoor@wspd0 llama.cpp]$ ls -lrt
total 20204
-rw-r--r-- 1 akapoor tsiusers 47860 Aug 5 10:07 AUTHORS
-rw-r--r-- 1 akapoor tsiusers 12356 Aug 5 10:07 CMakeLists.txt
-rw-r--r-- 1 akapoor tsiusers 4570 Aug 5 10:07 CMakePresets.json
-rw-r--r-- 1 akapoor tsiusers 6091 Aug 5 10:07 CODEOWNERS
-rw-r--r-- 1 akapoor tsiusers 8481 Aug 5 10:07 CONTRIBUTING.md
-rw-r--r-- 1 akapoor tsiusers 1078 Aug 5 10:07 LICENSE
-rw-r--r-- 1 akapoor tsiusers 257 Aug 5 10:07 Makefile
-rw-r--r-- 1 akapoor tsiusers 33656 Aug 5 10:07 README.md
-rw-r--r-- 1 akapoor tsiusers 5347 Aug 5 10:07 SECURITY.md
-rwxr-xr-x 1 akapoor tsiusers 21790 Aug 5 10:07 build-xcframework.sh
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 ci
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 cmake
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 common
-rwxr-xr-x 1 akapoor tsiusers 458746 Aug 5 10:07 convert_hf_to_gguf.py
-rwxr-xr-x 1 akapoor tsiusers 24353 Aug 5 10:07 convert_hf_to_gguf_update.py
-rwxr-xr-x 1 akapoor tsiusers 19106 Aug 5 10:07 convert_llama_ggml_to_gguf.py
-rwxr-xr-x 1 akapoor tsiusers 20291 Aug 5 10:07 convert_lora_to_gguf.py
drwxr-xr-x 6 akapoor tsiusers 4096 Aug 5 10:07 docs
drwxr-xr-x 28 akapoor tsiusers 4096 Aug 5 10:07 examples
-rw-r--r-- 1 akapoor tsiusers 1556 Aug 5 10:07 flake.lock
-rw-r--r-- 1 akapoor tsiusers 7243 Aug 5 10:07 flake.nix
drwxr-xr-x 5 akapoor tsiusers 4096 Aug 5 10:07 gguf-py
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 grammars
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 include
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 licenses
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 media
-rw-r--r-- 1 akapoor tsiusers 3319 Aug 5 10:07 model-rerun.py
drwxr-xr-x 3 akapoor tsiusers 4096 Aug 5 10:07 models
-rw-r--r-- 1 akapoor tsiusers 163 Aug 5 10:07 mypy.ini
drwxr-xr-x 3 akapoor tsiusers 4096 Aug 5 10:07 pocs
-rw-r--r-- 1 akapoor tsiusers 124786 Aug 5 10:07 poetry.lock
-rw-r--r-- 1 akapoor tsiusers 1336 Aug 5 10:07 pyproject.toml
-rw-r--r-- 1 akapoor tsiusers 616 Aug 5 10:07 pyrightconfig.json
-rw-r--r-- 1 akapoor tsiusers 551 Aug 5 10:07 requirements.txt
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 requirements
drwxr-xr-x 5 akapoor tsiusers 4096 Aug 5 10:07 scripts
drwxr-xr-x 2 akapoor tsiusers 4096 Aug 5 10:07 tests
drwxr-xr-x 17 akapoor tsiusers 4096 Aug 5 10:07 tools
-rwxr-xr-x 1 akapoor tsiusers 48962 Aug 5 10:07 tsi-pkg-build.sh
drwxr-xr-x 7 akapoor tsiusers 4096 Aug 5 10:07 vendor
drwxr-xr-x 11 akapoor tsiusers 4096 Aug 5 10:15 ggml-tsi-kernel
drwxr-xr-x 12 akapoor tsiusers 4096 Aug 5 10:24 build-fpga
drwxr-xr-x 3 akapoor tsiusers 4096 Aug 5 10:25 tsi-ggml
-rw-r--r-- 1 akapoor tsiusers 19509080 Aug 5 10:25 tsi-ggml-0.4.21.tz
-rwxr-xr-x 1 akapoor tsiusers 993 Aug 5 10:26 tsavorite-model-deployment.yaml
-rw-r--r-- 1 akapoor tsiusers 2403 Aug 5 10:33 run.log
drwxr-xr-x 2 akapoor tsiusers 8192 Aug 5 12:22 src
-rwxr-xr-x 1 akapoor tsiusers 9515 Aug 5 12:29 patch_tsavorite_mixed_precision_matmul.sh
-rwxr-xr-x 1 akapoor tsiusers 8091 Aug 5 12:37 patch_tsavorite_mixed_precision_matmul_v2.sh
-rwxr-xr-x 1 akapoor tsiusers 7066 Aug 5 12:38 patch_tsavorite_mixed_precision_matmul_v3.sh
-rwxr-xr-x 1 akapoor tsiusers 2341 Aug 5 12:47 patch_tsavorite_matmul_bf16_support.sh
-rwxr-xr-x 1 akapoor tsiusers 1848 Aug 5 12:49 patch_tsavorite_none_bf16_support.sh
drwxr-xr-x 5 akapoor tsiusers 4096 Aug 5 12:56 ggml
drwxr-xr-x 12 akapoor tsiusers 4096 Aug 5 13:22 build-posix
-rw-r--r-- 1 akapoor tsiusers 0 Aug 5 13:24 tsi-op.txt
-rw-r--r-- 1 akapoor tsiusers 11135 Aug 5 13:26 ggml_perf-all-shape.log
-rw-r--r-- 1 akapoor tsiusers 7866 Aug 5 13:26 ggml_op_shape_dtype_catalog.tsv
[akapoor@wspd0 llama.cpp]$ cat ggml_op_shape_dtype_catalog.tsv
Op Decision Count Result Src0 Src1

ADD SUPPORTED 46 f32 [2048,1,1,1] f32 [2048,1,1,1] f32 [2048,1,1,1]
ADD SUPPORTED 37 f32 [2048,512,1,1] f32 [2048,512,1,1] f32 [2048,512,1,1]
ADD SUPPORTED 42 f32 [2048,7,1,1] f32 [2048,7,1,1] f32 [2048,7,1,1]
CONT SUPPORTED 22 f16 [12288,64,4,1] f16 [12288,64,4,1] none [0,0,0,0]
CONT SUPPORTED 22 f16 [256,64,4,1] f16 [256,64,4,1] none [0,0,0,0]
CONT SUPPORTED 22 f32 [2048,512,1,1] f32 [64,32,512,1] none [0,0,0,0]
CONT SUPPORTED 22 f32 [2048,7,1,1] f32 [64,32,7,1] none [0,0,0,0]
CPY SUPPORTED 1 f16 [12288,64,1,1] f32 [12288,64,1,1] f16 [12288,64,1,1]
FLASH_ATTN_EXT REJECTED 22 f32 [64,32,1,1] f32 [64,1,32,1] f16 [64,12288,4,1]
GET_ROWS SUPPORTED 2 f32 [2048,1,1,1] f32 [2048,1,1,1] i32 [1,1,1,1]
GET_ROWS SUPPORTED 1 f32 [2048,1,1,1] f32 [2048,32003,1,1] i32 [1,1,1,1]
GET_ROWS SUPPORTED 2 f32 [2048,1,1,1] f32 [2048,7,1,1] i32 [1,1,1,1]
GET_ROWS SUPPORTED 2 f32 [2048,512,1,1] f32 [2048,512,1,1] i32 [512,1,1,1]
GET_ROWS SUPPORTED 1 f32 [2048,7,1,1] f32 [2048,32003,1,1] i32 [7,1,1,1]
GLU SUPPORTED 23 f32 [5632,1,1,1] f32 [5632,1,1,1] f32 [5632,1,1,1]
GLU SUPPORTED 17 f32 [5632,512,1,1] f32 [5632,512,1,1] f32 [5632,512,1,1]
GLU SUPPORTED 21 f32 [5632,7,1,1] f32 [5632,7,1,1] f32 [5632,7,1,1]
MUL_MAT REJECTED 22 f32 [12288,512,32,1] f16 [64,12288,4,1] f32 [64,512,32,1]
MUL_MAT REJECTED 22 f32 [256,7,32,1] f16 [64,256,4,1] f32 [64,7,32,1]
MUL_MAT REJECTED 22 f32 [64,512,32,1] f16 [12288,64,4,1] f32 [12288,512,32,1]
MUL_MAT REJECTED 22 f32 [64,7,32,1] f16 [256,64,4,1] f32 [256,7,32,1]
MUL_MAT SUPPORTED 44 f32 [2048,1,1,1] f32 [2048,2048,1,1] f32 [2048,1,1,1]
MUL_MAT SUPPORTED 23 f32 [2048,1,1,1] f32 [5632,2048,1,1] f32 [5632,1,1,1]
MUL_MAT SUPPORTED 40 f32 [2048,512,1,1] f32 [2048,2048,1,1] f32 [2048,512,1,1]
MUL_MAT SUPPORTED 19 f32 [2048,512,1,1] f32 [5632,2048,1,1] f32 [5632,512,1,1]
MUL_MAT SUPPORTED 44 f32 [2048,7,1,1] f32 [2048,2048,1,1] f32 [2048,7,1,1]
MUL_MAT SUPPORTED 21 f32 [2048,7,1,1] f32 [5632,2048,1,1] f32 [5632,7,1,1]
MUL_MAT SUPPORTED 44 f32 [256,1,1,1] f32 [2048,256,1,1] f32 [2048,1,1,1]
MUL_MAT SUPPORTED 37 f32 [256,512,1,1] f32 [2048,256,1,1] f32 [2048,512,1,1]
MUL_MAT SUPPORTED 44 f32 [256,7,1,1] f32 [2048,256,1,1] f32 [2048,7,1,1]
MUL_MAT SUPPORTED 2 f32 [32003,1,1,1] f32 [2048,32003,1,1] f32 [2048,1,1,1]
MUL_MAT SUPPORTED 2 f32 [32003,512,1,1] f32 [2048,32003,1,1] f32 [2048,512,1,1]
MUL_MAT SUPPORTED 46 f32 [5632,1,1,1] f32 [2048,5632,1,1] f32 [2048,1,1,1]
MUL_MAT SUPPORTED 37 f32 [5632,512,1,1] f32 [2048,5632,1,1] f32 [2048,512,1,1]
MUL_MAT SUPPORTED 42 f32 [5632,7,1,1] f32 [2048,5632,1,1] f32 [2048,7,1,1]
MUL SUPPORTED 52 f32 [2048,1,1,1] f32 [2048,1,1,1] f32 [2048,1,1,1]
MUL SUPPORTED 34 f32 [2048,512,1,1] f32 [2048,512,1,1] f32 [2048,1,1,1]
MUL SUPPORTED 43 f32 [2048,7,1,1] f32 [2048,7,1,1] f32 [2048,1,1,1]
NONE SUPPORTED 45 f32 [2048,1,1,1] none [0,0,0,0] none [0,0,0,0]
NONE SUPPORTED 44 f32 [2048,2048,1,1] none [0,0,0,0] none [0,0,0,0]
NONE SUPPORTED 44 f32 [2048,256,1,1] none [0,0,0,0] none [0,0,0,0]
NONE SUPPORTED 1 f32 [2048,32003,1,1] none [0,0,0,0] none [0,0,0,0]
NONE SUPPORTED 44 f32 [2048,5632,1,1] none [0,0,0,0] none [0,0,0,0]
NONE SUPPORTED 22 f32 [5632,2048,1,1] none [0,0,0,0] none [0,0,0,0]
RMS_NORM SUPPORTED 47 f32 [2048,1,1,1] f32 [2048,1,1,1] none [0,0,0,0]
RMS_NORM SUPPORTED 37 f32 [2048,512,1,1] f32 [2048,512,1,1] none [0,0,0,0]
RMS_NORM SUPPORTED 43 f32 [2048,7,1,1] f32 [2048,7,1,1] none [0,0,0,0]
ROPE SUPPORTED 22 f32 [64,32,1,1] f32 [64,32,1,1] i32 [1,1,1,1]
ROPE SUPPORTED 22 f32 [64,32,512,1] f32 [64,32,512,1] i32 [512,1,1,1]
ROPE SUPPORTED 22 f32 [64,32,7,1] f32 [64,32,7,1] i32 [7,1,1,1]
ROPE SUPPORTED 22 f32 [64,4,1,1] f32 [64,4,1,1] i32 [1,1,1,1]
ROPE SUPPORTED 21 f32 [64,4,512,1] f32 [64,4,512,1] i32 [512,1,1,1]
ROPE SUPPORTED 22 f32 [64,4,7,1] f32 [64,4,7,1] i32 [7,1,1,1]
SOFT_MAX SUPPORTED 22 f32 [12288,512,32,1] f32 [12288,512,32,1] f32 [12288,512,1,1]
SOFT_MAX SUPPORTED 22 f32 [256,7,32,1] f32 [256,7,32,1] f32 [256,64,1,1]
[akapoor@wspd0 llama.cpp]$

Summary by cubic

Enables generic mixed‑precision MAT_MUL offload by packing GGML inputs to F32 for the Triton TMU; outputs remain F32. Addresses FIR‑2152 by extending offload to F16, BF16, and quantized weights (block‑aligned, contiguous K) with tighter checks and clearer errors.

  • New Features

    • Offloads when A/B are F32/F16/BF16 or quantized types with to_float, if K is block‑aligned and nb0 matches the type size; result tensor stays F32.
    • Adds unified, K‑stride aware F32 pack and scatter helpers for both full and small‑N transpose paths, handling non‑contiguous strides and quantized blocks.
    • Improves capability cataloging: marks GGML_OP_NONE tensors with packable dtypes as SUPPORTED and records precise reasons (e.g., mul_mat shape/dtype supported or not).

  • Refactors

    • Replaces manual A/B copy loops with centralized helpers, validates nb0 vs. type size, and adds explicit error messages with tsi_cleanup() before abort.
    • Adds comments clarifying host‑side dequantization via GGML type traits before calling the F32 Triton MAT_MUL kernel.