fix(rag): scope chunk IDs to their source document

EL
4d ago
+89-7

PR Checklist

  • Commit follows the commit convention.

Bugs / Features

  • Related issue linked: fixes #1551.
  • Regression tests added.
  • Changeset added for @voltagent/rag.
  • Documentation: no guide change; this restores distinct identities in the existing multi-document API. The changeset records the generated-ID format change.

What is the current behavior?

StructuredDocument.chunk() concatenates per-document chunker output without scoping the IDs. Two one-chunk documents both return token-0; both link-graph entries refer to the same ID. A Map keyed by the returned IDs retains just one of the two chunks.

What is the new behavior?

Generate each final chunk ID from its source document ID, document position and final chunk ordinal, and use it for the existing document links. Explicit and generated document IDs both work; contents and metadata.docId are preserved. The document position also disambiguates repeated explicit document IDs. Generated links attach to the exact source node, so repeated document IDs group the two source chunks once each instead of cross-linking them to both nodes.

This is separate from #1526's graph-key handling and the individual chunker expansion fixes: this change only establishes identity at the multi-document collection boundary.

Validation

  • Unchanged built public @voltagent/rag export: two chunks, IDs ["token-0", "token-0"], keyed collection size 1. After this change: two distinct IDs, size 2, correct per-document links.
  • New public-interface regression suite: 4 failures / 2 controls passing before, 7 passing after. The original six regressions cover token, recursive, Markdown, generated document IDs, repeated-call stability, and empty input. A seventh regression for repeated explicit document IDs fails against the initial PR head and passes after the review update. A later review exposed duplicate source-node links: strengthened exact graph/node-link assertions fail against the previous head and pass with source-specific attachment; all 54 tests and package builds pass again.
  • pnpm exec vitest run --root packages/rag: 54 tests passing across the complete RAG package.
  • pnpm --filter @voltagent/rag build: CJS, ESM and declaration builds pass.
  • Biome, changeset Markdown formatting, git diff --check, and the actual lint-staged/commit-message hooks pass.

These checks use the real default tokenizer and document/chunker/link code. The keyed collection is a native local Map; no external model or vector database was used. Monorepo-wide builds/tests were not run for this isolated RAG change.

Prepared with AI assistance.


Summary by cubic

Fixes chunk ID collisions in StructuredDocument.chunk(), where chunks from different documents could share IDs (e.g., token-0), causing the returned collection and link graph to lose data.

  • Generates each chunk ID as <document-id>-<document-index>-chunk-<ordinal>, so distinct documents and repeated document IDs stay distinct.
  • Applies to explicit and generated document IDs; contents and metadata.docId are preserved.
  • Adds regression tests covering token, recursive, and markdown chunkers, generated IDs, repeated explicit IDs, repeated calls, and empty input.

Summary by CodeRabbit

  • Bug Fixes
    • Document chunks now have IDs tied to their source document and position, preventing chunks from different documents from sharing IDs—even when documents have duplicate explicit IDs.
    • Chunk IDs remain stable when unchanged content is processed again across token-based, recursive, and Markdown chunking.
    • Documents without a specified ID receive generated identities for their chunks. Empty documents continue to produce no chunks or links, and chunk metadata and link mappings remain associated with the correct document.