feat(rag): wire vector_store, BM25, and RRF params into retrieve backend (#913)
AD
3d ago+593+593+012345678901234567890123456789-45-45-01234567890123456789
Follow-up to #839. Closes #913.
Summary
Wires \ector_store, \m25_k1, \m25_b, and
rf_k\ from \RAGConfig\ / \OperatorSpec.params\ into the retrieve backend and query pipeline:
- IDL & Schema: Defined \enum VectorStore\ in \idl/rag.proto\ (imported in \idl/solutions.proto). Added \m25_k1, \m25_b,
rf_k, and \ector_store\ fields to \RAGRetrievalOptions\ and \RAGConfiguration. Bumped \idl/VERSION\ to 1.6.0 and updated \idl/SCHEMA_LOCK. - BM25 Parameterization: Parameterized \BM25Index::search(query, top_k, k1, b)\ to accept custom term-saturation (\k1) and length-normalization (\) values.
- RRF Parameterization: Parameterized \use_rankings(..., top_k, rrf_k)\ to honor custom smoothing constants (
rf_k). - Graph & Backend Wiring: Threaded \m25_k1, \m25_b, and
rf_k\ through \RAGGraphInputs\ in
ag_pipeline_graph.cpp\ and \RAGBackend::query\ / \RAGBackend::retrieve. - Retrieve Operator: Updated \RetrieveNode\ in \op_engine_backed.cpp\ to parse \ector_store, \m25_k1, \m25_b, and
rf_k, forwarding them into the \RAGQueryOptions\ request. - PGVECTOR Guard: Explicitly rejects \VECTOR_STORE_PGVECTOR\ with an honest error both at session configuration / query parsing and in \RetrieveNode.
- Tests:
- Added unit test in \ est_rag_fusion.cpp\ verifying custom
rf_k\ alters reciprocal rank decay. - Added test in \ est_solution_runner.cpp\ verifying \RetrieveNode\ rejects \VECTOR_STORE_PGVECTOR\ with honest cancellation.
- Added tests in \ est_advanced_modality_proto_abi.cpp\ verifying \VECTOR_STORE_PGVECTOR\ rejection and acceptance of custom BM25/RRF options on \RAGSearchRequest.
- Added unit test in \ est_rag_fusion.cpp\ verifying custom
Summary by cubic
Wires vector_store, bm25_k1, bm25_b, and rrf_k from RAGConfig and OperatorSpec.params into the retrieve backend, replacing hardcoded retrieval defaults. Closes #913 as a follow-up to #839.
- Adds
vector_store,bm25_k1,bm25_b, andrrf_ktoRAGConfigurationandRAGRetrievalOptions;VectorStoreenum moves fromsolutions.prototorag.proto; IDL bumped to 1.7.0. - Rejects
VECTOR_STORE_PGVECTORon-device (only USearch has an on-device path); unknown enum values are rejected as invalid. - Rejects non-finite
bm25_k1,bm25_boutside [0, 1], and negativerrf_kat both the ABI and retrieve operator boundaries. - RetrieveNode strictly parses
k,similarity_threshold,bm25_k1,bm25_b, andrrf_k, rejecting malformed values,k < 1, and out-of-range thresholds. - RRF accumulates in double and tie-breaks by id so a large
rrf_kcannot tie candidates; float params are formatted with the classic locale so they round-trip on any host. - Tests added for measurable BM25 parameter effects, custom RRF decay, pgvector rejection, and malformed-param rejection.
Summary by CodeRabbit
- New Features
- Added configurable BM25 ranking and reciprocal-rank fusion settings for RAG sessions and individual retrieval requests, with defaults preserved when overrides are omitted.
- Added vector-store selection for RAG retrieval. USEARCH is supported on-device; PGVECTOR requests are rejected as unsupported.
- Bug Fixes
- Retrieval requests now reject malformed, non-finite, or out-of-range ranking settings and unsupported or unknown vector-store selections with clear errors.