PR #1154 - RunanywhereAI/runanywhere-sdks

Description

Malformed UTF-8 in Pythonic arguments can survive serialization while changing the value: double-quoted strings retain literal surrounding quotes, and nested objects or arrays become strings. Repair the bytes before parsing so the decoded value and its JSON type are preserved.

Share the existing LLM UTF-8 sanitizer with the tool parser; reject overlong encodings, surrogate code points, and values above U+10FFFF while preserving valid Unicode.
Use replacement handling when serializing raw argument keys, validation errors, and tool-result error payloads.
Move the existing tool-result payload conversion into its internal result helper so malformed in-memory errors can be tested directly.

For example, value={"q":"<0xFF>"} now yields an object containing U+FFFD rather than a JSON-encoded string. A double-quoted value="<0xC3><0x28>" yields the string �( without extra quotes.

Fixes #1020.

Related: #948 is unmerged at this base; this change includes the same final argument-serialization safeguard and works independently. Default-format argument repair remains tracked in #1131.

Type of Change

Bug fix
Refactoring: share existing internal UTF-8 and result-payload helpers

Testing

Regression coverage for tagged LFM2 and bare Pythonic parsing, decoded strings, object/array types, schema validation, malformed argument keys, Unicode bounds, valid Unicode, validation errors, and tool-result error payloads.
Linux Debug protobuf-enabled build of the affected commons library and five test executables.
Tool parser binary: 33/33 cases passed.
CTest: 5/5 passed — tool parser, tool-calling run loop, tool-calling session proto, LLM proto service, and LLM stream proto.
Five mutation checks: removing pre-parse repair, argument replacement handling, validation-error replacement handling, tool-error replacement handling, or Unicode scalar bounds makes the corresponding regressions fail.
Changed-line formatting, new-header formatting, diff whitespace, PII-logging guard, and deprecated-surface gate.
Repository-wide formatting is not clean: clang-format 18.1.8 flags the same 80 existing source files as the clean base, plus an untracked generated defaults header. No newly flagged tracked file.

Baseline reproduction: unchanged production sources return RAC_ERROR_INTERNAL for malformed Pythonic arguments and throw type_error.316 while serializing validation errors containing an invalid tool-name byte.

The malformed tool-result test constructs the in-memory result directly because strict protobuf decoding rejects malformed string fields before prompt formatting.

Platform-Specific Testing

Native Linux Debug only. Apple, Android, Windows, and Web integration checks were not run locally.

Labels

Commons — shared native code in core/

Checklist

Self-review completed
Changes follow project style in the edited regions
Internal comments describe replacement behavior

Summary by cubic

Malformed UTF-8 in tool arguments previously changed values during serialization: double-quoted strings kept literal quotes and nested objects or arrays became strings. Repairing the bytes before parsing preserves the decoded value and JSON type.

Shares the LLM UTF-8 sanitizer with the tool parser, rejecting overlong encodings, surrogates, and values above U+10FFFF while preserving valid Unicode.
Applies replacement handling when serializing raw argument keys, validation errors, tool-result error payloads, and executor errors.
Rejects the entire call when repaired keys collide at any object depth, including inside arrays, instead of silently dropping an argument or returning a partial object; invalid JSON syntax still falls back to the raw-string value.
Extends Pythonic argument-key detection to non-ASCII bytes so malformed keys reach repair without loosening ASCII identifier and delimiter checks.
Moves tool-result payload conversion into an internal result helper so malformed in-memory errors can be tested directly.