fix(rag): keep oversized AST chunk IDs unique
PR Checklist
- The proposed commit message follows Conventional Commits.
Bugs / Features
- Related issue(s) linked. Reproduced directly on main.
- Tests for the changes have been added.
- Docs have been added or updated.
- A patch changeset for
@voltagent/raghas been added.
What is the current behavior?
When an oversized AST block is split by the token chunker, its output IDs add both the callback index and the current output length. Later AST blocks can reuse an ID already emitted by the split.
What is the new behavior?
Derive each split chunk ID from the number of chunks already emitted. The regression covers a large AST block and two subsequent small blocks.
Notes for reviewers
- The upstream fixture produces four unique IDs for five chunks. The patched regression also checks the exact sequence from
code-0throughcode-4. vitest run --typecheckpassed 48 RAG tests across 20 files.- Scoped TypeScript, changed-file Biome, changeset Prettier and whitespace checks passed.
- CommonJS, ESM and type declaration builds passed.
OpenAI Codex assisted with reproduction, implementation and local validation.
Summary by cubic
Fixes duplicate chunk IDs when token splitting expands an oversized AST block before other blocks. Split chunks now derive their ID from the number of chunks already emitted instead of combining the callback index with the output length, so later blocks can no longer reuse an ID from the split.
Adds a regression test for a large AST block followed by two smaller blocks, plus a patch changeset for @voltagent/rag.
Summary by CodeRabbit
- Bug Fixes
- Fixed duplicate code chunk IDs when a large syntax block is split into multiple chunks before other blocks. Chunk IDs now remain unique and sequential.