ggml-tsavorite: don't report OPU Target for ops with no kernel invocation

Closed
AK
3mo ago
+14-1

Summary

Found while reviewing GGML Perf Summary output enabled by FIR-2216/PR #158. Some ops with
TSI_KERNEL-RUN == 0 kernel launches were still labeled "OPU" in the Target column, which reads
as if they ran on Tsavorite hardware when they were actually computed via a direct call to
ggml's own CPU compute function.

Example (before this fix):

  Op               Target      Runs  TSI_KERNEL-RUN          Total us            Avg us
  SET_ROWS         CPU          728               0              4799              6.59
  GET_ROWS         OPU           15               0               369             24.60
  SOFT_MAX         OPU          110               0             78947            717.70
  ROPE             CPU          716               0             10128             14.15

SET_ROWS and ROPE correctly showed CPU, but GET_ROWS and SOFT_MAX -- going through the exact
same "no blob, call the plain CPU compute function directly" pattern in the dispatch switch --
showed OPU instead, despite also having 0 kernel runs.

Root cause

The per-node compute loop unconditionally marked every node it processed as
GGML_COMPUTE_BACKEND_TSAVORITE at the end of each iteration, regardless of whether a kernel
actually launched or the node was computed via the CPU-fallback path
(ggml_compute_forward_get_rows/ggml_compute_forward_soft_max/etc., called directly a few
lines earlier for op types with no kernel implementation).

ggml_tsavorite_internal_supports_op() accepts these op types identically (unconditional
return true for GET_ROWS, SOFT_MAX, SET_ROWS, ROPE, etc.), so the inconsistency in the example
above isn't explained there -- it's specifically the backend-label assignment at the end of the
compute loop.

Fix

Snapshot tsi_kernel_runs at the start of each node's processing and only report
GGML_COMPUTE_BACKEND_TSAVORITE if it actually increased during this node's compute; otherwise
report GGML_COMPUTE_BACKEND_CPU.

Validation

Verified via ollama (Makefile.sync temporarily pointed at this branch, merged with the
FIR-2238 perf_reset fix for a combined test): GET_ROWS, SOFT_MAX, and CONT (also affected, same
pattern) now correctly show CPU instead of OPU. ADD/MUL/RMS_NORM/MUL_MAT/GLU -- which do launch
real kernels -- are unaffected and still correctly show OPU. Confirmed the underlying
TSI_KERNEL-RUN magnitude for the reclassified ops is unchanged before/after this fix (this fix
only changes which Target bucket the existing numbers land in, not the numbers themselves).

Summary by cubic

Fixes mislabeling in GGML Perf Summary: nodes with zero TSI_KERNEL-RUN are no longer reported as OPU. Old behavior marked all processed nodes as OPU; new behavior reports OPU only if a kernel launched, otherwise CPU. This changes reporting only.

  • Captures tsi_kernel_runs at node start and sets node->ggml_compute_backend to TSAVORITE only if it increases; otherwise sets CPU.
  • Applies to ops without kernels (e.g., GET_ROWS, SOFT_MAX, CONT); kernel-backed ops remain OPU.
  • No change to execution path, run counts, or timings; only the Target classification is corrected.