feat(rag): wire vector_store, BM25, and RRF params into retrieve backend (#913)

AD
3d ago
+593-45

Follow-up to #839. Closes #913.

Summary

Wires \ ector_store, \m25_k1, \m25_b, and
rf_k\ from \RAGConfig\ / \OperatorSpec.params\ into the retrieve backend and query pipeline:

  • IDL & Schema: Defined \enum VectorStore\ in \idl/rag.proto\ (imported in \idl/solutions.proto). Added \m25_k1, \m25_b,
    rf_k, and \ ector_store\ fields to \RAGRetrievalOptions\ and \RAGConfiguration. Bumped \idl/VERSION\ to 1.6.0 and updated \idl/SCHEMA_LOCK.
  • BM25 Parameterization: Parameterized \BM25Index::search(query, top_k, k1, b)\ to accept custom term-saturation (\k1) and length-normalization (\) values.
  • RRF Parameterization: Parameterized \ use_rankings(..., top_k, rrf_k)\ to honor custom smoothing constants (
    rf_k).
  • Graph & Backend Wiring: Threaded \m25_k1, \m25_b, and
    rf_k\ through \RAGGraphInputs\ in
    ag_pipeline_graph.cpp\ and \RAGBackend::query\ / \RAGBackend::retrieve.
  • Retrieve Operator: Updated \RetrieveNode\ in \op_engine_backed.cpp\ to parse \ ector_store, \m25_k1, \m25_b, and
    rf_k, forwarding them into the \RAGQueryOptions\ request.
  • PGVECTOR Guard: Explicitly rejects \VECTOR_STORE_PGVECTOR\ with an honest error both at session configuration / query parsing and in \RetrieveNode.
  • Tests:
    • Added unit test in \ est_rag_fusion.cpp\ verifying custom
      rf_k\ alters reciprocal rank decay.
    • Added test in \ est_solution_runner.cpp\ verifying \RetrieveNode\ rejects \VECTOR_STORE_PGVECTOR\ with honest cancellation.
    • Added tests in \ est_advanced_modality_proto_abi.cpp\ verifying \VECTOR_STORE_PGVECTOR\ rejection and acceptance of custom BM25/RRF options on \RAGSearchRequest.

Summary by cubic

Wires vector_store, bm25_k1, bm25_b, and rrf_k from RAGConfig and OperatorSpec.params into the retrieve backend, replacing hardcoded retrieval defaults. Closes #913 as a follow-up to #839.

  • Adds vector_store, bm25_k1, bm25_b, and rrf_k to RAGConfiguration and RAGRetrievalOptions; VectorStore enum moves from solutions.proto to rag.proto; IDL bumped to 1.7.0.
  • Rejects VECTOR_STORE_PGVECTOR on-device (only USearch has an on-device path); unknown enum values are rejected as invalid.
  • Rejects non-finite bm25_k1, bm25_b outside [0, 1], and negative rrf_k at both the ABI and retrieve operator boundaries.
  • RetrieveNode strictly parses k, similarity_threshold, bm25_k1, bm25_b, and rrf_k, rejecting malformed values, k < 1, and out-of-range thresholds.
  • RRF accumulates in double and tie-breaks by id so a large rrf_k cannot tie candidates; float params are formatted with the classic locale so they round-trip on any host.
  • Tests added for measurable BM25 parameter effects, custom RRF decay, pgvector rejection, and malformed-param rejection.

Summary by CodeRabbit

  • New Features
    • Added configurable BM25 ranking and reciprocal-rank fusion settings for RAG sessions and individual retrieval requests, with defaults preserved when overrides are omitted.
    • Added vector-store selection for RAG retrieval. USEARCH is supported on-device; PGVECTOR requests are rejected as unsupported.
  • Bug Fixes
    • Retrieval requests now reject malformed, non-finite, or out-of-range ranking settings and unsupported or unknown vector-store selections with clear errors.