fix(rag): report original offsets for split fenced code

EM
2d ago
+35-0

PR Checklist

  • The proposed commit message follows Conventional Commits.

Bugs / Features

  • Related issue(s) linked. Reproduced directly on main.
  • Tests for the changes have been added.
  • Docs have been added or updated.
  • A patch changeset for @voltagent/rag has been added.

What is the current behavior?

When the parser returns no blocks and a fenced block exceeds maxTokens, CodeChunker returns token offsets relative to the fence body. With prose before the fence, using text.slice(chunk.start, chunk.end) can return prose while chunk.content contains code. The line and column metadata is also relative to the fence body.

What is the new behavior?

Translate the split chunks' start and end offsets to the original document and use those offsets to compute line and column metadata. Regression tests exercise LF and CRLF documents with prose before the fence.

Notes for reviewers

  • Both new regression cases fail on upstream main and pass with this patch.
  • vitest run --typecheck passed all 49 RAG tests in 20 files.
  • Scoped strict TypeScript, changed-file Biome, changeset Prettier and whitespace checks passed.
  • CommonJS, ESM and type declaration builds passed.

OpenAI Codex assisted with reproduction, implementation and local validation.

Summary by cubic

Fixes CodeChunker reporting token offsets relative to the fence body when a fenced code block exceeds maxTokens and is split by the fallback. Offsets and line/column metadata now point to the original document, so text.slice(chunk.start, chunk.end) returns the code content instead of possibly prose before the fence. Adds regression tests for LF and CRLF documents with prose before the fence.

Summary by CodeRabbit

  • Bug Fixes
    • Corrected position reporting for code chunks created when oversized code blocks are split as a fallback. Chunk offsets now point to the corresponding text in the original document, with line and column positions consistent across LF and CRLF line endings. This makes it easier to locate each chunk’s content in the source document.