fix(rag): stop chunker extras from overwriting format and sourceType

KU
6d ago
+68-1

PR Checklist

  • The commit message follows our guidelines (conventional commits)
  • Related issue(s) linked — none filed; found while reading buildMetadata
  • Tests for the changes have been added
  • Changesets have been added

What is the current behavior?

buildMetadata() sets format and sourceType from its own params (lines 20–23) and then does

Object.assign(meta, params.extra ?? {});

so anything in extra silently overwrites those two identifiers. That contradicts the contract stated a few lines above (base: ... core fields are not overridden by baseMetadata and the comment on baseMetadata).

It bites every chunker that delegates to an inner chunker and forwards the child chunk's metadata through extra, because the child metadata already carries its own format/sourceType:

  • html-chunker.ts:98 (extra: chunk.metadata from RecursiveChunker)
  • neural-chunker.ts:71 and neural-chunker.ts:96
  • slumber-chunker.ts:75
  • table-chunker.ts:66 and table-chunker.ts:109

Observed locally on main (72a46c76):

HTML: [{"id":"html-0","format":"text","sourceType":"sentence"},
       {"id":"html-1","format":"text","sourceType":"sentence-token"},
       {"id":"html-2","format":"text","sourceType":"sentence-token"}]
TABLE: 17 chunks, every one "format":"text" / "sourceType":"token"

So HtmlChunker — which declares format: "html", sourceType: "html" — reports "text" for all of its chunks, and TableChunker never reports "table" for prose it splits below the token limit. Consumers that filter or boost on metadata.format/metadata.sourceType get the delegate's answer, not the outer chunker's.

json-chunker.ts:89 already papers over this locally with const { sourceType: _ignored, ...chunkExtra } = ..., which is a good indication the precedence problem is real — it just gets fixed one chunker at a time instead of at the source.

What is the new behavior?

extra can still contribute any field it likes, but it can no longer overwrite the format / sourceType declared by the chunker that called buildMetadata:

Object.assign(meta, params.extra ?? {}, { format: params.format, sourceType: params.sourceType });

With this in place the per-chunker destructuring in json-chunker.ts becomes redundant (as would the proposed per-file fix in #1463). I deliberately left json-chunker.ts untouched so this PR stays scoped to the shared helper and does not overlap that PR's files.

Tests

Added packages/rag/src/__tests__/metadata.spec.ts covering the helper directly plus two delegating chunkers.

BEFORE (metadata.ts at main, new spec added)
   × buildMetadata > keeps chunker-specific extras without overriding the core identifiers 14ms
   × delegating chunkers > HtmlChunker reports html even though it delegates to the recursive chunker 14ms
   × delegating chunkers > TableChunker reports table for prose split below the token limit 5ms
 Test Files  1 failed | 18 passed (19)
      Tests  3 failed | 44 passed (47)

AFTER (with this fix)
 Test Files  19 passed (19)
      Tests  47 passed (47)

Baseline before adding the new spec was Test Files 18 passed (18) / Tests 43 passed (43) — none of the existing specs asserted metadata.format, which is why this stayed unnoticed.

Notes for reviewers

  • packages/rag is not in the test-packages matrix of pull-request.yml, so the rag specs never execute in CI. Worth considering separately from this PR.
  • Repro/verification ran on Node v24.21.0 with vitest 3.2.4 + js-tiktoken 1.0.21 (the only runtime dependency of @voltagent/rag), via vitest run src/__tests__ inside packages/rag.
  • AI assistance: this change was prepared with an AI coding assistant (Claude via Qoder). I reviewed every line of the diff and the test output above myself; the root cause and the call-site list were confirmed by reading the source, not generated.

fixes (issue): N/A — no issue filed for this yet.

Summary by cubic

Fixes buildMetadata so the caller-declared format and sourceType can no longer be overwritten by the extra metadata that delegating chunkers forward from their inner chunkers.

  • HtmlChunker and TableChunker were reporting the inner chunker's identifiers (e.g. "text" instead of "html"), breaking consumers that filter or boost on those fields.
  • Adds tests for buildMetadata and the affected delegating chunkers, with the TableChunker spec now asserting sourceType per prose and table chunk group; the per-chunker workaround in json-chunker.ts is redundant but left untouched to avoid overlapping another PR.

Summary by CodeRabbit

  • Bug Fixes
    • Chunk metadata now consistently reports the format and source type of the chunker that produced it, even when additional metadata contains conflicting values.
    • HTML and table chunking results preserve their own identifiers, while retaining other supplied metadata, including extra fields that do not conflict with those identifiers.