fix(rag): preserve table chunk source ranges

LJ
17h ago
+60-6

PR Checklist

  • The commit message follows the conventional commit guidelines.

Bugs / Features

  • Related issue linked: fixes #1467.
  • Regression tests added.
  • Docs added / updated (not needed: corrects existing source-range behavior).
  • Patch Changeset added for @voltagent/rag.

What is the current behavior?

TableChunker delegates prose to SentenceChunker and oversized tables to TokenChunker, but returns their substring-relative offsets. Prose after a table starts at zero, and split table chunks omit the table's start. A citation/highlighting consumer that slices the original document consequently selects unrelated text. Delegated line/column metadata also refers to the substring.

What is the new behavior?

Track each prose buffer's start and translate delegated prose/table offsets into original-document coordinates. Compute position metadata from the original document's line map, including the existing unsplit-table path. Contents, grouping and token budgets are unchanged.

This branch is based directly on upstream main, independently of #1466, and does not contain its recursive-ID fix.

Notes for reviewers

Validation on Windows with Node.js 24.19.0 and upstream baseline 72a46c76b4507e1f98b41d5050a92bd30587eb70:

  • On unchanged runtime, the 4 new regressions fail; the 2 existing table tests pass. Cases cover split/unsplit tables, surrounding prose, repeated table/prose blocks, LF and CRLF.
  • Fixed RAG suite: 47 tests pass across 18 files (vitest run --config packages/rag/vitest.config.mts packages/rag/src/__tests__).
  • Strict TypeScript check of both changed TypeScript files: passed.
  • Biome check of both changed TypeScript files: passed.
  • Actual package tsup build: CommonJS, ESM and declarations passed.
  • git diff --check: passed.
  • The package-wide tsc --project packages/rag/tsconfig.json --noEmit still reports 3 pre-existing errors in code-chunker.spec.ts and late-chunker.spec.ts; the identical errors also occur on untouched upstream with the same dependencies. Full package type checking is not claimed to pass.

Tests run the actual source with isolated repo-compatible tooling and the declared tokenizer dependency. Full monorepo tests, live models and vector-store integration were not run. CI has not been verified.

AI assistance: Codex authored, tested and inspected this scoped change under LJK-ikun's direction. Local automated validation is reported separately from human review; independent human testing is not asserted.