fix(commons): preserve tool calls with invalid utf-8
SH
14d ago+34+34+01234567890123456789-1-1-0123456789
Summary
Malformed UTF-8 in model-generated string arguments can make JSON serialization fail and drop an otherwise valid tool call. Replace invalid bytes with U+FFFD during LFM2/Pythonic argument serialization, and add regression cases for both tagged LFM2 and bare Pythonic parsing.
The branch is rebased onto main. The production change is one line; main's exception handling, buffer ownership, parser statuses, and format fallback remain intact.
Validation
- Focused tool-calling suite built from production sources with C++20 and warnings treated as errors: all 21 non-protobuf cases passed, including both new malformed-UTF-8 regressions. Six protobuf-only cases were compiled out of this local runtime check.
- Protobuf-enabled parser and test translation units compiled with warnings treated as errors.
- Mutation check: removing the replacement policy makes exactly the two new regression cases fail with the internal-error result; all other cases retain their previous results.
- Diff whitespace and added-line width checks passed.
Full platform integration remains unverified locally; the updated branch still requires CI validation. The repository formatter and commit hook runner were unavailable on this Windows host.
Summary by CodeRabbit
- Bug Fixes
- Tool calls with invalid UTF-8 in quoted arguments can now be serialized successfully, with invalid bytes replaced by the Unicode replacement character. Tool names and detected formats are preserved.