feat(semantic): support high-dimensional embeddings
Merged
CH
6mo ago+113+113+012345678901234567890123456789-5-5-0123456789
Closes #49
Summary
Raises the embedding dimension cap from 1024 to 4096 so larger models like
text-embedding-3-large (3072) can be indexed through the external backends.
The cache format and all defaults stay the same.
What changed
semantic_index.rs:MAX_DIMENSIONgoes from 1024 to 4096. Dimension
checking is consolidated into onevalidate_embedding_dimension()that runs
both when a backend returns a batch and when a cache is read back.- Tests: build, persist, reload, and search at 4096; reject 0 and 4097 on both
build and deserialization.
Why just the cap and one check
This PR intentionally stays small: a constant change plus a single boundary
check. Bigger questions like aggregate memory ceilings and corrupt-cache
hardening are left out so they can be handled on their own if you want them.
Validation
cargo fmt --checkcargo test -p agent-file-tools --test semantic_validation_testcargo test -p agent-file-tools semantic_index
Summary by cubic
Raises the embedding dimension cap to 4096 to support high‑dimensional models like text-embedding-3-large (3072) and common 4096‑d local models. Cache format and defaults are unchanged.
-
New Features
- Increase
MAX_DIMENSIONfrom 1024 to 4096. - Allow 4096‑d embeddings during build and read; reject 0 and >4096 with clear errors.
- Increase
-
Refactors
- Add
validate_embedding_dimension()and use it in batch validation and deserialization.
- Add