@FIR-2216 - llama.cpp: expose ggml perf-summary symbols across ollama's split-binary backend boundary

Merged
AK
3mo ago
+241-6

Summary

Ollama statically links llama-context.cpp into its Go binary via cgo, while the ggml core (ggml.c) and the Tsavorite backend are built into separately dlopen'd shared libraries (libggml-base.so, libggml-tsavorite.so). Several symbols the "GGML Perf Summary" table depends on are defined in ggml.c and were link-time unreachable from llama-context.cpp in that configuration:

  • ggml_perf_accumulate — undefined reference at link time (table never printed at all).
  • ggml_perf_log_open, ggml_perf_write_detailed_csv, ggml_backend_type — same issue, only surfaced once the detailed (Target/TSI_KERNEL-RUN column) table variant was exercised.

Also, ggml-tsavorite.cpp's existing #ifdef OLLAMA backend struct literal already references .profile, .graph_reserve, .buffer_size, .reset, but ggml-backend-impl.h didn't yet define those ggml_backend_i fields, and ggml_backend_log_profile_info() (declared in ggml-backend.h) had no definition — both needed for that file to even compile under -DOLLAMA.

Changes

  • include/llama.h, src/llama.cpp: add llama_backend_log_profile(); restore the ggml_backend_free(TSAVORITE) call in llama_backend_free().
  • ggml/include/ggml-backend.h, ggml/src/ggml-backend.cpp: add ggml_backend_log_profile_info().
  • ggml/src/ggml-backend-impl.h: add graph_reserve, buffer_size, reset, profile fields to struct ggml_backend_i.
  • ggml/src/ggml-tsavorite/ggml-tsavorite.cpp: expose ggml_perf_accumulate, ggml_perf_log_open, ggml_perf_write_detailed_csv, ggml_backend_type via ggml_backend_tsavorite_get_proc_address().
  • src/llama-context.cpp: add a shared llama_resolve_ggml_proc_address() helper plus #ifdef OLLAMA-gated wrappers (llama_perf_ggml_accumulate, llama_perf_ggml_log_open, llama_perf_ggml_write_detailed_csv, llama_perf_ggml_backend_type) that resolve these symbols dynamically at runtime through the backend registry. Native (non-OLLAMA) builds keep the original direct calls, unchanged.

Scope is intentionally llama.cpp-only. Ollama picks this up via a separate Makefile.sync commit-id bump once this merges -- no ollama-side changes in this PR.

Also included (found while investigating a separate, still-open multi_thread_enable:true intermittent crash in ollama, unrelated to the perf-symbol work above but touching the same file):

  • Fix a real data race in tsi_init_per_txe_state_once(): unsynchronized std::vector mutation (packed_args, device_free) on first use from concurrent callers.
  • Harden call_triton_matmul_full_packed(): its static descriptor buffers were populated with no synchronization before reaching the tsi_pack_mutex-protected dispatch call.

These are real, standalone concurrency fixes, but do not fully resolve the intermittent multi_thread_enable:true SIGSEGV in the Triton MAT_MUL dispatch path (still under investigation, tracked separately).

Validation

Tested in ollama (posix build, SDK_VERSION=0.4.24), via a temporary local sync pointed at this branch:

  • Native llama.cpp posix debug build compiles and links cleanly with all changes (sanity check independent of ollama).
  • Ollama posix debug build (GGML_PERF_DETAIL) — full detail table prints correctly, matching llama.cpp's own tsi-pkg-build.sh dev output format: 
    === GGML Perf Summary ===
  Op               Target      Runs  TSI_KERNEL-RUN          Total us            Avg us
  ADD              OPU          480        30496556            677744           1411.97
  MUL              OPU          490        30847744            653504           1333.68
  RMS_NORM         OPU           10          970154             14996           1499.60
  MUL_MAT          OPU          165        51087589          63340037         383879.01
  CONT             OPU          240        15149184             16538             68.91
  GET_ROWS         OPU           30         2201664              2113             70.43
  SOFT_MAX         OPU          240        15096192             27038            112.66
  GLU              OPU          240        15274774            696186           2900.78
    Verified via qwen:0.5b, HTTP 200, no crash.
  • Ollama posix release build (GGML_PERF_RELEASE) — plain customer-safe table prints correctly, no CPU/OPU breakdown: 
    === GGML Perf Summary ===
  Op                   Runs        Total us            Avg us
  ADD                   144          287062           1993.49
  MUL                   147          558433           3798.86
  RMS_NORM                3           37089          12363.00
  MUL_MAT               165        61081193         370189.05
  CONT                   72           16335            226.88
  GET_ROWS                9            2070            230.00
  SOFT_MAX               72           24200            336.11
  GLU                    72          545814           7580.75
    Verified via qwen:0.5b, HTTP 200, no crash.
  • Both tested with multi_thread_enable:false (the known-stable path); multi_thread_enable:true remains separately tracked as an open, intermittent issue unrelated to this fix.

Summary by cubic

Restores GGML perf-summary tables in ollama split-binary builds and adds a backend profile hook. Previously llama-context.cpp could not reach ggml_perf_*/ggml_backend_type and printed no tables; now runtime resolution restores detailed and release tables, and exports for detailed CSV are correctly guarded to prevent release build link errors. Addresses FIR-2216.

  • Dynamic perf symbols for ollama: ggml-tsavorite.cpp exposes ggml_perf_accumulate and ggml_backend_type in all perf modes, and exposes ggml_perf_log_open/ggml_perf_write_detailed_csv only under GGML_PERF_DETAIL; src/llama-context.cpp adds OLLAMA-only wrappers that resolve these at runtime with atomic caches and retry after late backend registration; native llama.cpp builds keep direct calls.
  • Backend profiling: adds ggml_backend_log_profile_info() and optional graph_reserve, buffer_size, reset, profile in struct ggml_backend_i. Introduces llama_backend_log_profile() and caches a single TSAVORITE backend; tracks when the profile hook finalizes the runtime to avoid double teardown in llama_backend_free().
  • Concurrency hardening in ggml-tsavorite: serializes first-use lazy init in tsi_init_per_txe_state_once() via an atomic flag plus mutex; serializes call_triton_matmul_full_packed() descriptor populate+dispatch; on all teardown paths, clears packed_args and all scalar_* per‑TXE vectors and resets the init flag to force clean reallocation on the next init.
  • Rollout: no migration required; ollama will pick this via Makefile sync.