fix(rag): detect top-level JSON arrays in auto format detection

TH
2d ago
+36-1

PR Checklist

  • The commit message follows our guidelines (conventional commits)
  • Related issue(s) linked
  • Tests for the changes have been added
  • Docs have been added / updated
  • Changesets have been added

What is the current behavior?

detectFormat() accepted { or [ as the opening character but only ever accepted } as the closing one:

if ((text.startsWith("{") || text.startsWith("[")) && text.endsWith("}")) {

A top-level JSON array ends with ], so it never entered the JSON.parse branch and was reported as "text". detectStrategy() in document/transformers.ts maps that result to a chunker, so with strategy: "auto" any document whose content is a JSON array — API response dumps, exported record sets, .json fixtures — went to RecursiveChunker instead of JsonChunker, losing the structural path: value chunks and the fields / samplePaths metadata.

Object payloads were unaffected.

What is the new behavior?

The brackets are paired, so arrays are detected too:

if (
  (text.startsWith("{") && text.endsWith("}")) ||
  (text.startsWith("[") && text.endsWith("]"))
) {

The existing try { JSON.parse(text) } catch still guards malformed input, so [unclosed] keeps falling through to the html / latex / code / table / markdown checks and still returns "text".

fixes (issue)

fixes #1458

Notes for reviewers

  • Scope is the array case only. I deliberately did not touch the table check below (/\n\|.+\|\n/ requires a newline before the first row, so a table at the very start of a document is also missed) — that is a separate bug and I would rather not bundle it.
  • Also noted in #1458 but not changed here: JsonChunker's format: "json" is overwritten by the format: "text" it inherits from the token chunker, because buildMetadata ends with Object.assign(meta, params.extra) and only sourceType is stripped from the inherited metadata.
  • New file packages/rag/src/__tests__/format-detector.spec.ts with four cases: array detection, object detection, malformed fall-through, and an end-to-end check that chunkByStrategy("auto", '[{"name":"alice"},{"name":"bob"}]') now yields metadata.fields === 2 and a chunk containing 0.name: alice.

Verified locally (Windows host, no network or API keys):

  • vitest run in packages/rag -> 47 passed across 19 files.
  • With format-detector.ts reverted to main, the array detection test and the end-to-end test fail (expected 'text' to be 'json', expected undefined to be 2); with the change they pass.
  • biome check clean on both changed files (the CRLF complaint when checking a raw Windows checkout disappears against the committed LF blob).

AI assistance: this fix and its tests were produced with an AI coding agent under my direction. The bug, the paired-bracket fix, the duplicate check and every test result quoted above were verified by running those commands against a checkout of main at 72a46c7.

Summary by cubic

Fixes detectFormat() so top-level JSON arrays are detected as JSON instead of "text". Previously the check accepted { or [ as an opening bracket but only } as a closing one, so arrays ending in ] fell through to RecursiveChunker under the auto strategy, losing JSON structural chunks and metadata. Brackets are now paired ({...} and [...]), and JSON.parse still guards malformed input.

fixes #1458

Summary by CodeRabbit

  • Bug Fixes
    • Automatic format detection now recognizes top-level JSON arrays, allowing them to be chunked as JSON. Malformed bracketed input continues to be treated as text.