fix(search): batch embedding requests in executeSearch to avoid N+1 HTTP fan-out

EP
3mo ago
+16-22

What

executeSearch embeds one string per request: embedText([q]) for the query, then embedText([content]) per result inside Promise.all. For N results that is N+1 concurrent single-input embeddings requests.

This batches them into one embedText([q, ...contents]) call, then slices the result apart.

Why

Every other embedText call site already batches. The contract is embedText(texts: string[]) across all providers (openai, transformers, ollama), and OpenAIEmbedding sends the whole array as one request. Ingestion relies on it:

  • src/lib/uploads/store.ts: embedText(queries)
  • src/lib/uploads/manager.ts: embedText(splittedText), embedText(pdfSplittedText), embedText(docSplittedText)

executeSearch is the only caller that fans out one string at a time, so this makes search embed the way ingestion already does.

It also unbreaks single-device backends. NPU/ANE servers (FastFlowLM, Apple ANE, llama.cpp-embedding) embed serially and drop concurrent connections (TCP RST or empty replies). The N+1 fan-out made them unusable for non-trivial searches; one batched request works.

Semantics

Identical: same > 0.5 filter, same Chunk shape, same try/catch fallback. Batch size is bounded by the SearXNG result count (~10-20).

Scope

One function in one file (baseSearch.ts). No config or API change.

Testing

  • tsc --noEmit: clean
  • prettier --check: passes
  • Verified against an OpenAI-compatible endpoint and a single-device NPU backend.

Summary by cubic

Batch embedding requests in executeSearch to replace N+1 single-input calls with one batched embedText([q, ...contents]). This reduces HTTP fan-out and fixes failures on single-device embedding backends.

  • Bug Fixes
    • Replaced per-result calls with one batched call and sliced embeddings to compute similarity.
    • Restores compatibility with NPU/ANE and llama.cpp-embedding servers that drop concurrent connections.
    • No behavior or API changes; change limited to src/lib/agents/search/researcher/actions/search/baseSearch.ts.