storage.cache_dir: keep rebuildable caches on another volume

AF
1h ago
+15438-852

Accepted agent delivery; branch-keyed search descriptor included.


Summary by cubic

Adds the storage.cache_dir user setting so rebuildable caches (indexes, views, semantic data, and reranker models) live on a separate volume instead of durable storage. When that volume is missing, unreadable, or identity-changed, all disk caches turn off with cache_dir_unavailable rather than falling back to the main disk, and aft clean-old-caches --yes removes legacy cache folders after confirmation.

  • The setting is honored from the user config base only; project values and harness overrides are dropped, an explicit cache_dir: null moves caches back to the storage root, and a relative path is refused under the setting's own name.
  • All cache artifact paths resolve through a cache-root-aware helper (cache_artifact_root) so AFT_STORAGE_DIR can no longer redirect them onto durable storage; the same resolved root feeds standing verifications, URL fetches, URL-cache expiry, and the stale reclaim-token sweep.
  • Key derivation composes with the durable owner, so a reader naming the cache root gets the same identity key as the durable root that owns it, and the configure read-only semantic arm reuses the already-resolved key for shared snapshots.
  • Version-floor and persisted-format refusals for cache artifacts are spelled under the durable root that owns them, so they are gated, listed, and cleared under that owner; the cache root never registers as a storage owner of its own.
  • Enrollment is restartable: the ownership marker is written and synced before the pending record, the staging lease is closed before the Windows rename, and a foreign folder that reuses a published path without a marker is treated as unpublished so doctor cleanup can drop the record.
  • First enrollment requires a separate device from durable storage and reports cache_dir_on_storage_device otherwise; symlinked roots are resolved once and pinned by device and inode, and a repointed link is refused.
  • On Windows, resolved cache paths drop the verbatim prefix and read-only SQLite stores are checked through their decoded file: URI.
  • Model downloads re-prove the enrolled cache root right before hf-hub receives the pathname, refusing with cache_dir_unavailable on a reroot.
  • The standing volume probe re-runs configure only when the check changed cache state, so a deferred storage.cache_dir edit never turns caches off mid-session; a repaired config re-enables caches even when its text changed, misspelled storage keys reject the config, and cache paths in responses are redacted under the cache root as well as the durable root.
  • Cleanup runs as a single grouped retention round; its deletion-protection census tolerates entries removed mid-walk, the cached health reply bounds cache-root lists, and process enumeration is bounded and fails closed so spaced executable paths and undecidable scans refuse rather than over-match. Daemons started before this change must be restarted before cleaning old storage.

RetriggerView in GreptileConfidence Score: 3/5

Fix URL downloads falling back to durable storage and stale probe updates before merging.

Findings

  1. P1 Downloads return to main disk ▶
  2. P1 Old probes overwrite reconnects ▶
  3. P2 Fresh config repairs report failure ▶

Summary

Adds user-only storage.cache_dir to move rebuildable caches onto another volume.

  • AFT puts rebuildable artifacts on a separately selected cache volume.
  • Cache roots stay tied to their owner as volumes and sessions change.
  • Operators can confirm removal of legacy cache folders under durable storage.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    U["User storage.cache_dir"] --> P["Check volume and ownership"]
    P -->|Available| C["Cache volume: indexes, views, models"]
    P -->|Unavailable| O["Disk caches off"]
    D["Durable storage"] --> M["Ownership, history, tasks and backups"]
    D --> L["Cleanup locks"]
    L --> X["Confirmed removal of old caches"]

Reviews (12) · Last reviewed commit: "mason: register views with their storage..." · Reviewed by Greptile