feat(semantic): support high-dimensional embeddings

Merged
CH
6mo ago
+113-5

Closes #49

Summary

Raises the embedding dimension cap from 1024 to 4096 so larger models like
text-embedding-3-large (3072) can be indexed through the external backends.
The cache format and all defaults stay the same.

What changed

  • semantic_index.rs: MAX_DIMENSION goes from 1024 to 4096. Dimension
    checking is consolidated into one validate_embedding_dimension() that runs
    both when a backend returns a batch and when a cache is read back.
  • Tests: build, persist, reload, and search at 4096; reject 0 and 4097 on both
    build and deserialization.

Why just the cap and one check

This PR intentionally stays small: a constant change plus a single boundary
check. Bigger questions like aggregate memory ceilings and corrupt-cache
hardening are left out so they can be handled on their own if you want them.

Validation

  • cargo fmt --check
  • cargo test -p agent-file-tools --test semantic_validation_test
  • cargo test -p agent-file-tools semantic_index

Summary by cubic

Raises the embedding dimension cap to 4096 to support high‑dimensional models like text-embedding-3-large (3072) and common 4096‑d local models. Cache format and defaults are unchanged.

  • New Features

    • Increase MAX_DIMENSION from 1024 to 4096.
    • Allow 4096‑d embeddings during build and read; reject 0 and >4096 with clear errors.
  • Refactors

    • Add validate_embedding_dimension() and use it in batch validation and deserialization.