fix(rag): assign unique IDs after recursive chunk expansion

LJ
17h ago
+23-4

PR Checklist

  • The commit message follows the conventional commit guidelines.

Bugs / Features

  • Related issue linked: fixes #1465.
  • Regression tests added.
  • Docs added / updated (not needed: restores distinct IDs without changing the API).
  • Patch Changeset added for @voltagent/rag.

What is the current behavior?

Splitting one paragraph into several sentence groups gives each group the same recursive-N ID. A mixed sentence/token expansion also reuses IDs, because the outer chunks array has not grown while flatMap expands the paragraph. The public reproduction in #1465 returns three different chunks with only one distinct ID.

What is the new behavior?

Assign final IDs after the complete paragraph expansion, using the existing chunk count plus each expanded chunk's index. Sentence groups and token fallback chunks receive distinct IDs within a call. Splitting and overlap behavior are unchanged.

Notes for reviewers

Validation on Windows with Node.js 24.19.0, against upstream 72a46c76b4507e1f98b41d5050a92bd30587eb70:

  • New regressions on unchanged runtime: 2 failed, with unique/total counts of 1/3 and 4/8; the existing 2 recursive tests passed.
  • Fixed RAG suite: 45 tests passed across 18 files (vitest run --config packages/rag/vitest.config.mts packages/rag/src/__tests__).
  • Biome check of both changed TypeScript files: passed.
  • Strict TypeScript check of both changed TypeScript files: passed.
  • Package tsup build: CommonJS, ESM and declaration builds passed.
  • git diff --check: passed.
  • The package-wide tsc --project packages/rag/tsconfig.json --noEmit reports 3 pre-existing errors in code-chunker.spec.ts and late-chunker.spec.ts. The identical errors reproduce on the untouched upstream checkout with the same dependencies. They are outside this fix. This is not a claim that the full package type check passes.

Dependencies were installed in an isolated tool directory using versions compatible with the repo; tests execute the actual source. No implementation was recreated and no runtime collaborator was patched. Full monorepo tests and live model/vector-store integration were not run. CI has not been verified.

AI assistance: Codex authored, ran and inspected this focused change under LJK-ikun's direction. The report distinguishes local automated validation from human review; it does not assert independent human testing.

Summary by cubic

Fixes duplicate recursive-N IDs in @voltagent/rag when one paragraph expands into multiple chunks. IDs are now assigned after the full paragraph expansion, so sentence groups and token fallback chunks each get a distinct ID. Splitting and overlap behavior are unchanged; regression tests added.

Summary by CodeRabbit

  • Bug Fixes

    • Recursive document chunking now assigns unique, sequential IDs to expanded sentence and token chunks, preventing identifier collisions across multiple texts.
    • Improved consistency when chunks are created through different recursive expansion paths.

  • Tests

    • Added coverage confirming that recursively expanded chunks receive distinct IDs under constrained token limits.