fix(semantic): neutralize image data URIs before remote embedding requests

Draft
CI
2d ago
+122-1

Linked issue

Approved issue: #817

What changes

embed_texts sends chunk text verbatim as the embeddings input. VLM-backed
embedding deployments — NVIDIA's current nemotron-*embed* models, for
example — parse data:image/<mime>;base64 anywhere in an input string as an
image part and reject the whole request when VLM serving is disabled:

HTTP 400 image inputs require VLM serving to be enabled on this server

Source code legitimately builds such strings (image_url=f"data:image/png;base64,{...}",
screenshot helpers, docstrings naming the format), so one chunk killed its
batch and, with it, the entire semantic build — repeatedly, from chunk 0, every
session.

This change neutralizes the data:image/ prefix to data-uri:image/ in the
OpenAI-compatible request body only:

  • the persisted chunk text is unchanged, so search snippets and provenance keep
    the original bytes;
  • fastembed and ollama payloads are unchanged, since neither interprets data
    URIs;
  • text/plain, application/json, audio/* data URIs and the uppercase form
    are left alone — they already embed successfully against the same deployment,
    so rewriting them would only perturb vectors;
  • the helper borrows when the prefix is absent, so the common row costs one
    substring scan and no allocation.

The trigger is a literal, case-sensitive prefix and the backend does not
validate the payload, so a plain str::replace is enough; no regex is needed
on the request path.

embed_texts is the single request builder for both build batches and
interactive query embeds, so both are covered.

How it was checked

  • cargo fmt --check: clean.
  • Unit tests (image_data_uri_tests, 7 passed / 0 failed): borrowed
    pass-through, source-line template literal, multiple occurrences per row,
    docstring mention without a payload, non-image data URIs untouched, uppercase
    untouched, and a serialized body assertion that no data:image/ survives.
    Run in the debug profile: cargo test -p agent-file-tools --lib image_data_uri_tests.
    cargo test --release cannot build this crate's lib tests at this commit
    because storage_retention::test_volume_uuid_guard is #[cfg(debug_assertions)];
    that is unrelated to this change.
  • End-to-end against https://integrate.api.nvidia.com/v1 with
    nvidia/nemotron-3-embed-1b, on a one-file repository containing
    f"data:image/png;base64,{payload.decode('ascii')}", via aft index:
    • stock 0.62.0: kind=semantic status=gap usable=false reason="openai compatible request failed (HTTP 400 Bad Request): {\"message\": \"image inputs require VLM serving to be enabled on this server\" ...}", exit 2;
    • patched: kind=semantic status=success result_count=4, exit 0.
  • Real corpus, both roots that triggered the report, same NVIDIA backend, built
    through aft index into the same storage the plugin uses:
    • /mnt/s/ComfyUI: kind=search status=success result_count=1222,
      kind=semantic status=success result_count=20651 (the stock binary failed
      this root at batch 134/326 with the 400);
    • /go/career-ops: kind=search status=success result_count=3455,
      kind=semantic status=success result_count=3850.
      Exit 0, no build-breaker suspensions recorded afterwards.

Notes for review

  • No configuration knob: the rewrite is semantically neutral for an embedder and
    applies only to text that no longer matches the data-URI grammar.
  • Synapse and Ollama payload builders are untouched; if a future backend is
    known to parse data URIs, the helper is one call away.

AI assistance disclosure: this submission was produced with an AI coding agent
(DeepSeek V4.1 Flash driving an autonomous agent session). The agent reproduced
the failure end-to-end, wrote the patch and tests, and ran the verification
commands listed above; a human operator reviewed the diagnosis and directed the
approach. Review the diff with that in mind.

Summary by cubic

Fixes semantic embedding failures where VLM-backed deployments (e.g. NVIDIA's nemotron embed) reject chunks containing data:image/ prefixes as image inputs, returning HTTP 400 and breaking the whole build.

  • Rewrites data:image/ to data-uri:image/ only in the OpenAI-compatible request body; persisted chunk text, fastembed and ollama payloads, and non-image data URIs stay byte-identical.
  • The rewrite borrows when the prefix is absent, so the common row costs one substring scan and no allocation.
  • Adds unit tests covering borrowed pass-through, multiple occurrences, docstring mentions, non-image URIs, and serialized request body.