fix(commons): preserve tool calls with invalid utf-8

SH
14d ago
+34-1

Summary

Malformed UTF-8 in model-generated string arguments can make JSON serialization fail and drop an otherwise valid tool call. Replace invalid bytes with U+FFFD during LFM2/Pythonic argument serialization, and add regression cases for both tagged LFM2 and bare Pythonic parsing.

The branch is rebased onto main. The production change is one line; main's exception handling, buffer ownership, parser statuses, and format fallback remain intact.

Validation

  • Focused tool-calling suite built from production sources with C++20 and warnings treated as errors: all 21 non-protobuf cases passed, including both new malformed-UTF-8 regressions. Six protobuf-only cases were compiled out of this local runtime check.
  • Protobuf-enabled parser and test translation units compiled with warnings treated as errors.
  • Mutation check: removing the replacement policy makes exactly the two new regression cases fail with the internal-error result; all other cases retain their previous results.
  • Diff whitespace and added-line width checks passed.

Full platform integration remains unverified locally; the updated branch still requires CI validation. The repository formatter and commit hook runner were unavailable on this Windows host.

Fixes #868
Fixes #1021

Summary by CodeRabbit

  • Bug Fixes
    • Tool calls with invalid UTF-8 in quoted arguments can now be serialized successfully, with invalid bytes replaced by the Unicode replacement character. Tool names and detected formats are preserved.