fix(rag): preserve decimal sentence boundaries

WI
15h ago
+78-3

The sentence splitter treats every period as a boundary, so 2.5 and $3.99 become separate sentences. This preserves a period between ASCII digits while retaining the existing punctuation rules for dotted words, ellipses and sentences without intervening spaces. Source offsets and token counts use the complete sentence.

Refs #1542 (sentence-splitting half only). The MarkdownChunker offsets are being handled separately in #1543.

I added regressions for the utility and its SentenceChunker, RecursiveChunker and SemanticChunker callers, plus a patch changeset.

Validation on Node 24.19.0:

  • All 64 rag runtime tests pass; the new tests fail on the unchanged utility.
  • Package CJS/ESM/declaration build and changed-file Biome checks pass.
  • Independent checks of both built public exports pass.
  • Standalone package typecheck still reports three errors in the existing code-chunker and late-chunker tests. I reproduced the same diagnostics with the original utility; no errors point to the changed source or new tests.

This does not change abbreviation, Unicode-number or leading-dot-number handling. No dependencies or lockfile changed.

Summary by cubic

Fixes the sentence splitter so decimals like 2.5 and prices like $3.99 are no longer split at the period.

Previously every period was treated as a boundary. Now a period between ASCII digits keeps the number intact, while dotted words, ellipses, and sentences without intervening spaces behave as before. Source offsets and token counts use the complete sentence; abbreviation, Unicode-number, and leading-dot-number handling are unchanged. This covers the sentence-splitting half of #1542. Adds regressions for the utility and its SentenceChunker, RecursiveChunker, and SemanticChunker callers, plus a patch changeset for @voltagent/rag.

Summary by CodeRabbit

  • Bug Fixes
    • Sentence splitting now preserves decimal numbers and prices, such as 2.5 and $3.99, within their sentences.
    • Sentence-based chunking maintains the expected text and token counts for sentences containing decimals.