#1523 fix(rag): slice recursive overlap chunks by whole tokens, not characters

PR Checklist

Please check if your PR fulfills the following requirements:

• The commit message follows our guidelines: https://voltagent.dev/docs/community/contributing/#commit-convention

Bugs / Features

• Related issue(s) linked
• Tests for the changes have been added
• Docs have been added / updated
• Changesets have been added https://voltagent.dev/docs/community/contributing/#creating-a-changeset

What is the current behavior?

RecursiveChunker's overlap smoothing treats overlapTokens as a character count:

const overlapContent = next.content.slice(0, overlapTokens);

.slice(0, n) takes characters, so overlapTokens: 3 produces 3-character, mid-word fragments. Actual output on "First part is here. Second part follows soon. Third part concludes." (maxTokens 6, overlapTokens 3):

{"label":"recursive-overlap","content":"Fir"}
{"label":"recursive-overlap","content":" he"}
{"label":"recursive-overlap","content":" pa"}
{"label":"recursive-overlap","content":" Se"}
{"label":"recursive-overlap","content":" so"}

Each fragment is 1-2 tokens and starts or ends mid-word, so the overlap chunks add noise to the embedding store instead of context.

What is the new behavior?

The overlap slice uses sliceByTokenRange over the tokenizer's own tokens (whole tokens only), matching how TokenChunker's overlap option already works. The same input now yields whole-token overlap chunks (e.g. "First part is", exactly 3 tokens). This keeps the existing design (standalone overlap chunks sourced from the next chunk's head) and only corrects the unit.

Tests: new spec asserts every overlap chunk is a whole-token prefix of the following chunk with exactly overlapTokens tokens (fails on the old code: expected 2 to be 3). Full @voltagent/rag suite: 48/48 passing. biome check clean on the touched files.

Notes for reviewers

No linked issue - happy to open one first if preferred. The adjacent duplicate-ID behavior in the same expansion path is already covered by open PR #1466 and is intentionally out of scope here.

---

Summary by cubic

Fixes the RecursiveChunker overlap smoothing so overlapTokens is treated as a token count, not a character count, preventing mid-word fragments like "Fir" or " he" from entering the embedding store.

• Overlap chunks are now sliced as whole-token prefixes of the following chunk, matching how TokenChunker's overlap option already works.
• Fractional overlapTokens values are rounded up to avoid duplicating the next chunk.
• Adds specs asserting overlap chunks are whole-token prefixes with exactly overlapTokens tokens, plus a patch changeset for @voltagent/rag.

Summary by CodeRabbit

• Bug Fixes
  • Recursive text chunks now overlap by complete tokenizer tokens rather than character counts.
  • Fractional overlap settings are rounded up to the next whole token, while overlap content remains limited to the following chunk.