@FIR-1856 - llam.cpp:Revert Graph Split Optimization Due to Introduce…
Qwen model which was crashing earlier and also GLU, SOFT_MAX not showing OPU
[akapoor@wspd0 llama.cpp]$ gdb --args build-posix/bin/llama-cli-original -p "My cat's name is" -m /proj/rel/sw/ggml/models/qwen2-0_5b-instruct-q5_k_m.gguf --device tSavorite -c 12288 --temp 0.0 --n-predict 10 --repeat-penalty 1.5 -b 1024 --top-k 50 --top-p 0.9 --repeat-last-n 5 --no-warmup --no-conversation
GNU gdb (GDB) Rocky Linux 8.2-20.el8.0.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from build-posix/bin/llama-cli-original...done.
(gdb) run
Starting program: /proj/work/akapoor/llama-cpp-may16-sdk-r.0.4.5/llama.cpp/build-posix/bin/llama-cli-original -p My\ cat's\ name\ is -m /proj/rel/sw/ggml/models/qwen2-0_5b-instruct-q5_k_m.gguf --device tSavorite -c 12288 --temp 0.0 --n-predict 10 --repeat-penalty 1.5 -b 1024 --top-k 50 --top-p 0.9 --repeat-last-n 5 --no-warmup --no-conversation
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
warning: File "/proj/local/gcc-13.3.0/lib64/libstdc++.so.6.0.32-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
add-auto-load-safe-path /proj/local/gcc-13.3.0/lib64/libstdc++.so.6.0.32-gdb.py
line to your configuration file "/users/akapoor/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/users/akapoor/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
TSI deploy yaml=tsavorite-model-deployment.yaml txe_count=1 multi_thread_enable=0
[New Thread 0x7fffed652700 (LWP 1743991)]
[New Thread 0x7fffece51700 (LWP 1743992)]
[New Thread 0x7ffdea44f700 (LWP 1743993)]
[New Thread 0x7ffde9c4e700 (LWP 1744011)]
[Thread 0x7ffde9c4e700 (LWP 1744011) exited]
[New Thread 0x7ffde9c4e700 (LWP 1744012)]
My cat's name is[New Thread 0x7ffde8935700 (LWP 1744014)]
[New Thread 0x7ffde0f03700 (LWP 1744015)]
[New Thread 0x7ffd9680f700 (LWP 1744016)]
Mimi. She has 8 whiskers.
llama_perf_sampler_print: sampling time = 100.64 ms / 15 runs ( 6.71 ms per token, 149.04 tokens per second)
llama_perf_context_print: load time = 3739.14 ms
llama_perf_context_print: prompt eval time = 2122.23 ms / 5 tokens ( 424.45 ms per token, 2.36 tokens per second)
llama_perf_context_print: eval time = 5025.86 ms / 9 runs ( 558.43 ms per token, 1.79 tokens per second)
llama_perf_context_print: total time = 8872.52 ms / 14 tokens
=== GGML Perf Summary ===
Op Target Runs TSI_KERNEL-RUN Total us Avg us
ADD CPU 12325 0 30724 2.49
ADD OPU 2208 2392 2736788 1239.49
MUL OPU 2254 2442 1646798 730.61
RMS_NORM OPU 2254 2254 1798780 798.04
MUL_MAT CPU 39821 0 65900274 1654.91
CONT OPU 2208 0 172578 78.16
RESHAPE CPU 10848 0 1602 0.15
VIEW CPU 10936 0 1152 0.11
VIEW OPU 2208 0 501 0.23
PERMUTE CPU 7139 0 1059 0.15
PERMUTE OPU 2208 0 222 0.10
TRANSPOSE CPU 3451 0 479 0.14
GET_ROWS OPU 138 0 2587 18.75
SET_ROWS CPU 7758 0 5114 0.66
SOFT_MAX OPU 1104 0 139482 126.34
ROPE CPU 8681 0 26221 3.02
GLU OPU 1104 1196 3022226 2737.52
[Thread 0x7ffdea44f700 (LWP 1743993) exited]
[Thread 0x7fffece51700 (LWP 1743992) exited]
[Thread 0x7fffed652700 (LWP 1743991) exited]
OPU Profiling Results:
Profiler disabled
[Thread 0x7ffde9c4e700 (LWP 1744012) exited]
[Thread 0x7ffd9680f700 (LWP 1744016) exited]
[Thread 0x7ffde0f03700 (LWP 1744015) exited]
[Thread 0x7ffde8935700 (LWP 1744014) exited]
[Inferior 1 (process 1743918) exited normally]
Missing separate debuginfos, use: yum debuginfo-install brotli-1.0.6-3.el8.x86_64 cyrus-sasl-lib-2.1.27-6.el8_5.x86_64 keyutils-libs-1.5.10-9.el8.x86_64 krb5-libs-1.18.2-30.el8_10.x86_64 libcom_err-1.45.6-5.el8.x86_64 libcurl-7.61.1-34.el8_10.2.x86_64 libidn2-2.2.0-1.el8.x86_64 libnghttp2-1.33.0-6.el8_10.1.x86_64 libpsl-0.20.2-6.el8.x86_64 libselinux-2.9-8.el8.x86_64 libssh-0.9.6-14.el8.x86_64 libunistring-0.9.9-3.el8.x86_64 libxcrypt-4.1.1-6.el8.x86_64 openldap-2.4.46-20.el8_10.x86_64 pcre2-10.32-3.el8_6.x86_64 zlib-1.2.11-26.el8.x86_64
(gdb)