@FIR-1834 - Reduce GGML Graph Splits for Tsavorite Backend — Enable L…
NOT READY for REVIEW
Summary — Graph split optimization (Tsavorite + CPU)
Reduced excessive GGML scheduler graph splitting by fixing backend assignment for view ops in ggml_backend_sched_split_graph().
Previously, view ops (and in some cases their source tensors) were being forced onto the CPU backend, which polluted backend assignment and caused frequent Tsavorite ↔ CPU switching. Since a new split is created whenever the backend changes, this resulted in a large number of small, fragmented splits.
With this change, view ops are kept neutral during Pass‑2 expansion (no forced CPU assignment), allowing backend propagation to follow the intended scheduler behavior. This significantly reduces backend switching and results in fewer, larger graph splits without any functional impact.
Observed Results (Original vs With Fix)
Tiny-Llama-v0.3-FP32-1.1B-F32.gguf → 266 → 90 splits
tinyllama-vo-5m-para.gguf → 98 → 34 splits
smolVLM-256M.gguf → 247 → 187 splits
smolVLM-500M.gguf → 386 → 130 splits
SakanaAI-TinySwallow-1.5B-Instruct-F32.gguf → 341 → 229 splits
google-gemma-3-1b-it-F32.gguf → 368 → 212 splits
Consistent reduction observed across models, with significant improvements in cases that previously exhibited heavy backend ping-pong.
Test Command
gdb --args build-posix/bin/llama-cli-original
-p "My cat's name is"
-m
--device tSavorite
-c 12288 --temp 0.0 --n-predict 10
--repeat-penalty 1.5 -b 1024
--top-k 50 --top-p 0.9
--repeat-last-n 5
--no-warmup --no-conversation
Validated on Tiny-Llama-v0.3-FP32-1.1B-F32.gguf.
Further testing is in progress across additional models to check for regressions and continue comparing split behavior.