fix(llamacpp): keep streaming after an invalid UTF-8 sequence
run_decode_loop in llamacpp_backend.cpp decides how much of the buffered token bytes it can pass on by running each byte through Utf8State, a Hoehrmann UTF-8 DFA, and only cutting where the state is back at 0. That DFA's reject state is 1, and every transition out of 1 leads back to 1 (the whole s1 row of the table is 1). So once the model emits a single invalid sequence, the state never returns to 0 for the rest of the generation. valid_upto stays 0 and every later byte piles up in partial_utf8_buffer. Nothing streams again until the end-of-generation flush, and the stop-sequence check inside if (valid_upto > 0) never runs again.
I reproduced this with rac_llm_llamacpp_generate_stream on SmolLM2-135M-Instruct Q4_K_M (max_tokens 120, temperature 2, top_k 60000 so nearly the whole vocab is in play, seed 5). The model emits \xE5\xE5\xBF early on:
| chunks streamed | bytes in the last chunk | tokens generated with stop "Firefox" | |
|---|---|---|---|
main | 15 | 565 of 655 | 120 (the max_tokens limit) |
| this PR | 115 | 13 | 25 |
On main the text is still cut at "Firefox", because the stop-sequence wrapper in rac_llm_llamacpp.cpp finds it in the flushed text, but the model keeps decoding to max_tokens first. Seed 4 shows the same stall later in the run (61 chunks, then 317 bytes at once). The total bytes are identical before and after for every seed I ran, so only the timing of delivery changes, not the text.
What this changes: the DFA struct and its per-byte loop are removed, and valid_upto comes from rac::tokens::incomplete_utf8_tail, the helper the VLM funnel's Utf8Assembler is built on (#1296 uses it for the LLM funnels too). It only holds back an unfinished character at the very end of the buffer, so there's no state to get stuck in. The stop-window flush uses the same helper for its cut, so a run of stray continuation bytes can't pin the window either (the old loop there backed over every continuation byte).
Things worth knowing:
- Invalid bytes now go out with the chunk they arrive in instead of only in the final flush. They were already reaching callers through that flush on
main, so this doesn't add a new kind of output. #1296, which is still open, makes the LLM funnels replace them with U+FFFD. - I only saw invalid sequences with
top_keffectively off. With the defaulttop_kof 40, the 18 runs I tried (temperatures 1.5 to 8) on this small model never produced one, so how often this hits real chats depends on the model and the sampling settings. - I didn't show a built-in stop string (the chat-template ones) being missed, only the caller's stop. Both sit in the same skipped check.
Testing: there's no engine-level test I could add without a model, so I verified this with a throwaway probe linked against rac_backend_llamacpp (macos-debug, other backends off), using the numbers above, before and after the change. I didn't build the other SDKs locally, so CI covers those.
I'm a college freshman and used Claude Code to read the DFA table and write the probe. The numbers in the table come from runs I watched, and I checked the diff myself.
Summary by CodeRabbit
- Bug Fixes
- Improved UTF-8 handling in streamed responses. Characters split across incoming text chunks are kept intact before output, helping prevent incomplete or malformed characters from appearing in generated text. Stop-sequence detection and output also continue to operate on complete UTF-8 chunks, preserving readable text when a stop sequence occurs near a character boundary.