fix(llm): keep streamed LLM chunks valid UTF-8

AY
16h ago
+48-3

Fixes #1202.

The LLM streaming funnels forward whatever bytes the engine hands them, so a chunk that ends or starts mid-character goes out as an invalid proto string. #1202 assumed llama.cpp was safe because run_decode_loop buffers split characters, and I repeated that in the issue thread. It isn't safe. There are two ways it emits invalid UTF-8:

  • User stop sequences. The wrapper in rac_llm_llamacpp_generate_stream holds back a rolling window and emits stop_window.size() - user_stop_max_len bytes, with no check for a character boundary. Any stop sequence plus non-ASCII output is enough.
  • max_tokens landing mid-character. The end-of-generation flush in run_decode_loop appends the partial bytes it was holding and sends them.

I ran rac_llm_llamacpp_generate_stream directly with a throwaway probe (not in this PR), using stories260K.gguf and a GBNF grammar that forces the output to be 你好世界 repeated:

optionschunksinvalid UTF-8example
no stop sequence120E4 BD A0
stop "ZZZZZ"1212E4, then BD A0 E5, then A5 BD E4
max_tokens=411E4 BD A0 E5
max_tokens=511E4 BD A0 E5 A5

This breaks consumers rather than just rendering badly. With split chunks fed through rac_llm_generate_stream_proto on main, the C++ protobuf parser rejects all 4 events in the new test, including the terminal one.

What this changes: llm_stream_token_callback and the lifecycle stream_token_callback now pass the filtered text through Utf8Assembler, then sanitize_utf8, using the same assembler as the VLM funnels. Split characters are rejoined, and a partial character still held when the stream ends is dropped. sanitize_utf8 handles bytes that can never become valid, replacing them with U+FFFD as the unary path already does. Doing this in the funnels also covers QHexRT and NeuRT, which I couldn't inspect, so those no longer depend on an answer to my question in the issue.

Things worth checking in review:

  • A dangling tail differs between the two result paths. The stream drops it, as #1202 suggested, while the unary result turns it into U+FFFD through sanitize_utf8.
  • A chunk the assembler holds back still adds its tokens_in_delta, and in the lifecycle funnel it also sets the production timestamps. The handle funnel's first-token event still waits for the first delivered text, because its comment ties TTFT to what the user observes. The count only matters for engines with no get_stream_token_counts, and from the source tree that looks like NeuRT, though I can't see what it reports for tokens_in_delta.

What this doesn't fix: the RAG pipeline's trampoline and the OpenAI server's stream callback take engine chunks directly, and both pass user stop sequences through, so on llama.cpp they can still see split chunks. I didn't test either one. Backing emit_bytes up to a character boundary in the llama.cpp wrapper would cover them, but I kept that out of this PR.

Testing: I added test_stream_chunks_are_valid_utf8 to test_llm_proto_service.cpp, with a mock engine that emits a split character, an invalid byte and a dangling tail. It passes with this change (85 checks, 0 failures). With llm_module.cpp put back to main it failed (85 checks, 5 failures), which I ran before adding the output_tokens check, and that check fails on its own against the first commit of this PR (85 checks, 1 failure). There's no test for the timing move. The test only goes through the proto funnel, so the same change in the handle-based funnel isn't exercised. I also didn't run the real model through the funnels end to end, only through the engine call above. I built only that test (macos-debug, backends off) and the llama.cpp backend, so CI covers the SDK builds.

I'm a college freshman and had Claude Code help me trace the stream paths and write the probe. I read the diff and the probe output myself, and getting llama.cpp wrong in the #1202 thread was my miss.

Summary by cubic

Fixes #1202 by ensuring LLM streaming chunks are always valid UTF-8. llama.cpp could emit mid-character splits when user stop sequences or max_tokens landed on a partial byte, breaking proto parsers downstream.

Streamed tokens now pass through the same Utf8Assembler and sanitize_utf8 pipeline used by VLM funnels. Split characters are rejoined, a dangling tail at stream end is dropped, and bytes that can never be valid are replaced with U+FFFD.

  • Adds a mock-engine test covering split characters, invalid bytes, and a dangling tail.
  • Token counts now apply before assembly, so chunks the assembler holds back still count toward usage and max_tokens on engines without get_stream_token_counts; the lifecycle first-token timestamp moves earlier too.
  • The RAG pipeline trampoline and OpenAI server stream callback pass engine chunks through unchanged, so they may still see split UTF-8 on llama.cpp.

Summary by CodeRabbit

  • Bug Fixes
    • Improved text handling in streamed responses when characters are split across incoming chunks, so valid multibyte characters display correctly.
    • Invalid UTF-8 sequences are replaced with a replacement character, and incomplete trailing sequences are no longer shown as dangling text.
    • Streamed output token counts now account for text retained after filtering, including characters that are completed across multiple chunks.