fix(rag): map Markdown chunks to original source ranges

TO
11h ago
+175-23

PR Checklist

  • The commit message follows the commit convention

Bugs / Features

  • Related issue linked
  • Regression tests added
  • Docs added / updated (internal fix; source-span behavior described below)
  • Patch changeset added

What is the current behavior?

MarkdownChunker reports offsets relative to each extracted block, not the original Markdown. In #1542's example, both section chunks start at 0, so slicing the document by those ranges returns the heading instead of the section content.

What is the new behavior?

Addresses only the MarkdownChunker offsets half of #1542. The sentence-splitting/decimal half is left to @widechaos. This PR does not close the whole issue.

  • Parse the original Markdown and track source offsets for retained content code units. This avoids ambiguous text searches when blocks repeat.
  • Map both child-chunk boundaries back to the document, including stripped list/quote prefixes and joined paragraph lines.
  • Rebuild metadata.position against the original document line map.
  • Preserve original code indentation and Unicode rather than globally running normalizeText before parsing. Parsing supports LF, CRLF and CR line endings. Extracted paragraph/list/quote lines still use trimmed, space-joined presentation; code content uses LF between lines.

For a plain paragraph, markdown.slice(start, end) is its content. For a chunk spanning joined lines or list/quote items, the range is the source span: its slice can include original line breaks and internal Markdown markers that are absent from the extracted chunk content. It is not a claim that transformed content is a contiguous verbatim substring.

Tests

Five new regression cases cover the reported section ranges, repeated content with leading whitespace/CRLF/decomposed Unicode, multiline paragraphs/lists/quotes, token-split list content (including a repeated marker), and fenced code at both whole-block and token-split sizes.

  • All five new cases fail on unmodified main; the two original Markdown tests pass.
  • With the fix: full @voltagent/rag suite passes, 52 tests across 19 files.
  • Biome passes for both changed TypeScript files.
  • tsc --noEmit still reports the same three baseline errors in code-chunker.spec.ts and late-chunker.spec.ts. Compared the complete diagnostics with unmodified main: no new errors.

Notes for reviewers

Related: #1542. No changes to the sentence splitter or CodeChunker, and no dependency on #1534. Source mapping uses UTF-16 code-unit offsets, matching JavaScript slicing and the child chunker ranges. Existing tokenizer offset limitations remain outside this change.

Summary by cubic

Fixes MarkdownChunker so chunk start/end ranges and position metadata refer to the original Markdown source instead of each extracted block. Previously both section chunks in #1542 started at 0, so slicing the document returned the heading instead of the section content. This covers only the Markdown offsets half of #1542; the sentence-splitting/decimal half is left out and the issue is not closed.

  • For joined paragraph lines and list/quote items, the range is the source span, so it can include original line breaks and Markdown markers absent from the extracted chunk content; a plain paragraph slices exactly to its content.
  • Parsing uses the raw source instead of normalizing first, so CRLF/CR/LF endings, indentation, and Unicode offsets are preserved, and the position map is rebuilt against the original line map.
  • Closing fences are only recognized as complete raw lines (at most three leading spaces, at least three backticks, trailing spaces/tabs only); fence-like code lines with an info suffix no longer terminate the code block or promote code headings, and indented closing fences still end the block so later sections are not swallowed.
  • RecursiveChunker maps oversized later sentence groups to their real source origin instead of the group start.
  • Adds regression tests covering section ranges, repeated content, multiline paragraphs/lists/quotes, token-split list items, fenced code, indented and info-suffixed closing fences, and later sentence groups.

Summary by CodeRabbit

  • Bug Fixes
    • Markdown chunk ranges and position metadata now correspond more accurately to the original source, including content with different line endings, Unicode characters, lists, quotes, and fenced code blocks. Line positions and source-text correspondence are also more reliable when content spans sections or is split into smaller chunks, helping preserve the connection between generated chunks and their source text.

Follow-up validation

Two additional regressions were reproduced and fixed: raw indented closing fences must still end the code block; and RecursiveChunker's oversized later sentence groups were returning token offsets relative to the group without its origin. A small prerequisite correction in recursive-chunker.ts now adds the actual group-content origin before Markdown maps the range. No sentence-regex/decimal change. Synthetic join separators now map to the preceding source line end rather than inside the next stripped prefix. The token-split list test uses exact, untrimmed slice equality. Both new regressions fail on the prior PR head and pass after the fix. Full rag suite: 54/54 tests; same three baseline typecheck errors, no new diagnostics.

Fence predicate follow-up

Closing fences are now matched on the raw line, with at most three leading spaces, at least three backticks, and only trailing spaces/tabs. Fence-like code lines with an info suffix no longer terminate the Markdown block or promote code headings into section headings. Full rag suite: 55/55 tests, unchanged baseline typecheck diagnostics. The pre-existing tokenizer Unicode offset issue remains outside this PR and overlaps existing #1510; no duplicate tokenizer fix is included.