#162 @JIRA-2258 - Sync tsisw/llama.cpp to upstream commit 1f368f354 via consolidated Tsavorite patch

Final result: the dense-style gemma4:12b tag now fully works on the Tsavorite
backend — coherent output, matching the CPU backend, zero alignment warnings. The
MoE-style e2b/e4b tags still hit a separate, unrelated upstream loading bug —
not fixed by this sync, and not something a Tsavorite-side fix can address.

gemma4:e2b (MoE-style, 2012 tensors) — fails to load, upstream bug, backend-independent.

done_getting_tensors: wrong number of tensors; expected 2012, got 601

Traced via gdb to a thrown exception inside upstream's generic
llama_model_loader::done_getting_tensors() safety check — well past architecture
recognition and Gemma4-specific hparam/tensor-declaration loading, unrelated to the
Tsavorite backend. Directly confirmed backend-independent: re-ran with --device none (pure CPU, no Tsavorite backend involved) and got the identical silent
exit-1 failure. Independently confirmed, not just this PR's own finding:
ggml-org/llama.cpp#27349 reports
the same error class on gemma4:e4b. Not fixable in this PR; tracked as a follow-up to
check against later upstream commits.

gemma4:12b (dense-style, 667 tensors) — now fully works. Loads, dispatches real
OPU compute, produces coherent output matching the CPU reference (exact captured
generation: **"Luna0 vs CPU's "Luna", directly compared with a --device none
run — the **/trailing-0 is normal Tsavorite-vs-CPU formatting divergence, not
corruption), zero alignment warnings, exit 0. Getting here required finding and fixing
three real bugs in ggml-tsavorite.cpp:

1. Root cause: Gemma4's per-layer layer_output_scale weight is a genuine
   scalar tensor (ggml shape {1,1,1,1}), applied via ggml_mul(cur, out_scale)
   at the end of every FFN block — unique to Gemma4's architecture
   (src/models/gemma4.cpp, LLM_TENSOR_LAYER_OUT_SCALE), which is why Gemma3,
   Qwen2.5, and TinyLlama never hit this. Confirmed via direct value tracing on the
   real model: both operands are correct at the moment of kernel dispatch, but the OPU
   kernel writes nothing at all for this exact degenerate broadcast shape (ne10==1).
   Could not reproduce in isolation despite exhaustive attempts matching the exact
   shape/magnitude/dtype/op-sequence — a genuine kernel/runtime gap, not something
   fixable by adjusting this file's dispatch code. Fixed by computing this one
   degenerate shape directly on CPU (trivial for a scalar) instead of dispatching to
   the kernel; every other broadcast shape keeps using the OPU path unchanged.
2. Chunked elementwise-op (ADD/MUL/SUB/DIV) tail corruption: a non-32-float-multiple
   chunk width dispatched repeatedly (≥2×) back-to-back corrupts the tail of the first
   result. Fixed by padding each dispatch's length to the next 32-float multiple.
3. RMS_NORM multi-row corruption (found and fixed, confirmed not what Gemma4
   actually triggers — Gemma4 does issue multi-row RMS_NORM during prefill, but its
   256-wide rows are already a clean 32-multiple, so the buggy condition never fires;
   kept since it's a real, independently-reproducible bug).

Also fixed, pre-existing and unrelated to Gemma4's corruption specifically:
ggml_backend_tsavorite_buffer_type_get_alignment() returned 32 instead of
TSI_TVU_MEM_ALIGN (128) — confirmed identical in the old fork.

Reviewer feedback (cubic-dev-ai) addressed: the three fixes above are now scoped
to F32 only (they'd have misread F16's 2-byte elements as 4-byte floats — no tested
model uses F16 for these ops, so this closes a real gap without changing any verified
behavior); added join_all_workers() before each writeback as insurance against a
kernel variant dispatching asynchronously in a way not visible from this file;
ggml_backend_tsavorite_free() now frees/resets the padding scratch buffers instead of
leaving stale pointers across a second backend init/free cycle in the same process;
consolidated-patch.patch regenerated so it doesn't silently reintroduce the original
get_alignment() bug on the next sync; the new scalar-broadcast test now checks
ggml_backend_supports_op() before compute so it skips cleanly on TMU-only builds
instead of hitting an unhandled path.

An earlier hypothesis in this investigation attributed the alignment warnings to
Gemma4's embedding_length/head_count = 3840/16 = 240 (not a 128-byte multiple).
That was wrong — direct tracing of the model's real tensor shapes showed head_dim
is actually 256 (16 query heads, 8 grouped KV heads), already clean. Corrected in
GEMMA4-VALIDATION-SUMMARY.md
rather than left in place.

Every fix is regression-tested clean: byte-identical output and zero new alignment
warnings on all 4 standard posix models plus both Qwen K-quant controls (below), before
and after. New fast regression tests added:
examples/simple/simple-chunked-repro.cpp,
examples/simple/simple-scalar-broadcast-2d-repro.cpp.

Full detail, exact commands, and the tracing methodology in
GEMMA4-VALIDATION-SUMMARY.md.

Confirmed on real tsisim hardware, not just posix: akapoor re-ran Gemma4-12b (and
all 4 previously-tested models) on real tsisim against this PR's packaged FPGA-target
build — see
TSISIM-VALIDATION-SUMMARY.md
for the full writeup. This exercises the _host_new kernel variants (the FPGA-target
branch of the dispatch code every fix in this PR touches), not just posix's _host —
closing the gap that all of today's earlier testing was posix-only. tsisim's CPU output
matches posix's CPU output byte-for-byte; Tsavorite's one-token divergence from CPU is
the same already-documented minor decoding variance, not corruption. All 4
previously-tested models reproduce byte-identical output to their original tsisim
baseline. One tsisim-specific finding worth flagging so it isn't mistaken for a code
issue: tsisim needs user_dram_size_gb: 30 at txe_count: 20, not this PR's new 16GB
default — 16GB is correct and sufficient on posix (Gemma4-12b included); tsisim's real
per-TXE memory footprint is simply higher. This is a deployment-yaml tuning value on the
tsisim box, not something this PR needs to change.

Follow-up, not blocking this PR: report the true-scalar-broadcast kernel gap to
whoever owns TXE/OPU kernel compilation — the CPU fallback here is a correct, safe
workaround, not a fix at the kernel level.

CMakeLists.txt
common/CMakeLists.txt
common/common.h
common/log.h
common/sampling.cpp
docs/build.md
examples/CMakeLists.txt
examples/gguf/CMakeLists.txt
examples/gguf-hash/CMakeLists.txt
examples/lookup/CMakeLists.txt
examples/simple-chat/CMakeLists.txt
examples/simple/CMakeLists.txt
examples/simple/mat-mul-tsi-test-case.cpp
examples/simple/simple-backend-tsi.cpp
ggml/CMakeLists.txt
ggml/include/ggml-backend.h
ggml/include/ggml.h
ggml/include/ggml-tsavorite.h
ggml/src/CMakeLists.txt
ggml/src/ggml-backend.cpp
ggml/src/ggml-backend-impl.h
ggml/src/ggml-backend-reg.cpp
ggml/src/ggml.c
ggml/src/ggml-cpu/CMakeLists.txt
ggml/src/ggml-cpu/ggml-cpu.c
ggml/src/ggml-impl.h
ggml/src/ggml-tsavorite/CMakeLists.txt
ggml/src/ggml-tsavorite/ggml-tsavorite.cpp
ggml/src/ggml-tsavorite/include/TestModel.h
.github/CODEOWNERS
.gitmodules
include/llama.h
model-rerun.py
pocs/CMakeLists.txt
README.md
src/llama-context.cpp
src/llama-context.h
src/llama.cpp
src/llama-impl.h
tests/CMakeLists.txt
tools/cli/cli.cpp
tools/cli/CMakeLists.txt
tools/CMakeLists.txt
tools/completion/CMakeLists.txt
tools/completion/completion.cpp
tools/server/server.cpp
tsavorite-model-deployment.yaml

Migrated: the old single-file tools/main/main.cpp CLI (upstream renamed/restructured
this into tools/completion/, tools/cli/, hooking into tools/server/server.cpp).

Not just diff-based porting — these would have shipped broken if this sync had relied
on classification/hunk-porting alone:

• ggml_backend_buffer_i/ggml_backend_i positional struct literal misalignment
  (upstream inserted new fields mid-struct)
• ggml_compute_forward_reshape/permute/transpose calls removed (upstream consolidated
  these into inline no-ops)
• ggml-backend-reg.cpp backend registration — a genuine classification error from
  earlier in this sync; would have shipped a compiles-but-never-registers Tsavorite
  backend (only caught via --list-devices showing none, at runtime)
• -DLLAMA_BUILD_APP=OFF (new upstream unified binary breaks the fpga link otherwise)
• fit_params = false under GGML_TSAVORITE (new upstream device-memory-probing
  feature double-initializes the Tsavorite backend — confirmed via gdb to segfault)
• common_perf_print() explicitly calling llama_perf_context_print() (upstream moved
  perf-stat printing to a new function that never called the old one, silently dropping
  the Tsavorite perf table)
• fpga build/packaging: skip tests/ for the fpga target (matches old fork's actual
  behavior), add the SDK runtime lib path to the global linker flags
  (FPGA_RUNTIME_LIB_DIR), fix tsi-ggml packaging's llama-cli→llama-completion
  rename and its .so symlink/SONAME chain
• update_one_tsavorite_deployment_yaml() in tsi-pkg-build.sh didn't preserve
  multi_thread_enable from the existing yaml (every other flag was preserved) —
  confirmed pre-existing in the original Tsavorite code, fixed here since we were
  already touching this function

Added apply_consolidated_patch(): detects whether the patch is already applied
(checks for ggml-tsavorite.cpp) before attempting to apply it — mirrors the existing
one-time-setup pattern already used for the blob-creation venv. Verified as a true
no-op on an already-synced tree (byte-identical rebuild).

Day-to-day development (editing ggml-tsavorite.cpp, llama-context.cpp, or any other
tracked file) is completely unaffected — it's a normal git add/commit/push/PR, exactly
like before this sync. consolidated-patch.patch only matters again at the next big
upstream sync. To make that cheaper:

• UPSTREAM_BASE_COMMIT (repo root) — tracks the exact upstream commit
  consolidated-patch.patch corresponds to (previously undocumented anywhere).
• tsi-pkg-build.sh regenerate-patch — diffs that commit against current HEAD
  (excluding ggml-tsi-kernel, tsi-pkg-build.sh, and this PR's own evidence
  directory) and overwrites consolidated-patch.patch. Full next-sync workflow
  documented above do_regenerate_patch() in the script. Caught and fixed two real
  bugs in this helper while testing it (it initially swept this PR's own evidence docs
  into the patch, then a too-broad fix nearly dropped docs/build.md's real content) —
  see commit history.

• tools/server/server.cpp is touched (shared by llama-server/llama-cli/router mode).
  Necessary, not incidental: upstream's llama-cli now spawns llama-server
  in-process, so the Tsavorite cleanup hook has to live where the backend is actually
  torn down, which is now here. Minimal change (two ggml_backend_cleanup() calls next
  to existing llama_backend_free() calls), inert when GGML_TSAVORITE is off.
• LLAMA_OPENSSL kept OFF (preserves current behavior); the old LLAMA_CURL toggle was
  dead code (upstream deprecated it to a warning-only shim; the real control was always
  tsi-pkg-build.sh's own -DLLAMA_CURL=OFF) and was dropped.
• Tsavorite builds skip cli/server/ui entirely — matches the fork's existing
  pattern, not a new departure.

• A struct-initializer field-count mismatch in ggml.c's ggml_new_tensor_impl()
  (positional init supplies 3 values for a 4-field struct region — harmless today only
  because the shifted values happen to be 0)
• A GGML_TSAVORITE=OFF build break in ggml-cpu/CMakeLists.txt's fpga guards
  (references a variable only set inside if(GGML_TSAVORITE) — never hit in practice)