feat(agentenv): collect superseded runner volumes

Merged
BN
6d ago
+6969-256

Base is main, merged into the branch; the branch carries #618 on top of it. #618 was merged into chore/preloop-vm-hex-sha2, the branch #665 had already merged from, so its provider work never reached main (there is no crates/preloop-vm/src/agentenv/volume.rs in main) — this PR is where it lands, so it must not be rebased away. No dependency, lockfile or policy file changes (the supply-chain guard's dedicated-PR rule is unaffected).

What this adds

#618 publishes one content-addressed read-only volume per distinct runner tree and offers preloop volumes list|prune, but nothing removes a superseded volume on its own: every bundle change leaves a full runner tree behind. This adds the automatic pass and preloop volumes gc to run or preview exactly the same plan by hand.

The policy implemented

A plan is a pure function of what AgentENV lists, what is in use and each volume's age (agentenv::gc::plan_gc, unit-tested without a server). Per volume, in order:

  • keep — a live sandbox mounts it (mounted); this provider published and remembers it (live, while it is mountable: a volume AgentENV reports as failed or uploading is kept only until its grace period is over and then reclaimed as unusable, since the engine refuses to mount it and would otherwise stay stuck on it); the caller named it (--keep on the CLI); a populate in this process is writing it, or writing a published volume of the same digest (populating); it is younger than the grace period (young); nothing recorded its age (unknown-age); it is the newest superseded volume of its kind, kept for a rollback (previous); or no kind in the policy owns it (unmanaged).
  • delete — a published volume that is unmounted, past the grace period and beyond the rollback depth (superseded); a staging volume whose populate is gone (abandoned); a volume AgentENV reports as something other than ready, once it is past the grace period (unusable — this engine refuses to mount such a volume, so a start stuck on a failed one can populate again).

Defaults and knobs: PRELOOP_VOLUMES_GC=auto|off (default auto), PRELOOP_VOLUMES_KEEP_PREVIOUS (1), PRELOOP_VOLUMES_GC_GRACE_SECS (3600), PRELOOP_VOLUMES_GC_INTERVAL_SECS (21600). auto is the default because nothing else reclaims a superseded volume and the cost of being wrong is bounded: the provider re-derives a tree's fingerprint and re-checks the volume before reuse, so a volume deleted here is populated again (~1 minute) on the next start that needs it, not a failure. off disables the automatic passes only; volumes gc and prune are explicit acts and keep working.

Triggers: a background pass after a populate (a cold start or a bundle change, not on every start, spaced by a five-minute gap), a periodic pass every PRELOOP_VOLUMES_GC_INTERVAL_SECS, and preloop volumes gc [--keep <DIR>]... [--dry-run]. Each of the three has a test.

Three decisions a reviewer should sanity-check

  1. The pass deletes exactly what it classified, not "everything outside a keep set". prune_volumes lists inside the call and deletes anything not named in keep; for an automatic pass that would let another engine publish a volume between the plan and the deletion and have it deleted although no policy ever looked at it. The pass calls delete_volumes (added on the base branch) with the plan's delete set, so the plan, --dry-run and reality agree.
  2. In-flight protection is a record, not a lock. populate_volume records its staging volume (name, digest, host, created) in $PRELOOP_HOME/agentenv-volume-staging.json before creating it, and a Drop guard removes the record when the populate ends. A pass therefore keeps every staging volume in flight and every published volume of that digest without taking the per-tree populate lock — holding one for the length of a pass would serialize collection behind a ~1 minute populate. A populate another process is running is covered by the grace period (its record is seconds old); the module documents that window, and that a volume deleted inside it fails one start rather than corrupting anything.
  3. A pass always unions in what its own provider published, read right after its own listing, so an automatic pass cannot strand the engine it runs in even with an empty caller list.

Tests

  • cargo test -p preloop-vm --lib agentenv::gc — 35 passed: every keep reason and delete reason, the grace boundary, rollback depths 0/1/2, per-kind rollback with a plugged-in fake kind, unmanaged/unknown-kind volumes, an empty listing, a volume created in the future, plan determinism, the rendered table, and the staging registry's round-trip, merge, drop and stale-record rules.
  • cargo test -p preloop-vm --test agentenv_volume_gc — 11 passed against the fake aenv the provider's contract tests use: a mounted volume comes back in use and survives; a deleted volume is populated again on the next start (the memo revalidates the name) and its index entry goes with it; the rollback depth keeps the newest superseded volume and reclaims the older one; an abandoned staging volume is reclaimed while a fresh one is kept; a populate in flight keeps its staging volume; a dry run plans and deletes nothing; a named volume is never deleted; a volume the server reports as never-mountable is reclaimed; a plain start collects a superseded volume with nobody asking (the pass a populate triggers); and the timer collects one on a host where nothing ever starts, with the period shortened to 50 ms for the test.
  • cargo test -p preloop-vm -p preloop-cli -p preloop-orchestrator — 558 passed, 0 failed (233 + 8 + 150 + 14 + 88 + 37 + 11 + 17 across the binaries).
  • cargo clippy --locked -p preloop-vm -p preloop-cli -p preloop-orchestrator --all-targets -- -D warnings — clean. cargo fmt --all -- --check — clean.

Not verified

No live AgentENV was reachable from this machine, so the HTTP 409: volume is reserved by sandbox … refusal is exercised only through the fake's equivalent message and no end-to-end run against a real server was possible. The multi-engine case (two engines on one AgentENV) is reasoned about and documented rather than tested: a pass keeps its own published volumes, mounted volumes, anything young and its own in-flight stages, but an idle volume another engine published is a candidate once it is past the grace period, and that engine repopulates it instead of failing. Every deletion is logged at info with its name, size, kind and reason so this is visible after the fact.

Notes on the mechanics

The push went over https://github.com/preloopdev/preloop.git with gh's credential helper, because the github ssh remote hung twice; it used --no-verify, and the only installed pre-push hook in this checkout is the git-lfs shim (the repository's own push gate is not installed here), so nothing that would have run was silenced. CI is the gate.

Also verified by hand against the fake aenv, outside the test suite: preloop volumes gc --keep <bundle> --dry-run printed the plan and deleted nothing, and the same command without --dry-run deleted exactly the two volumes the plan named, logged both at info with name/size/kind/reason, dropped the deleted names from the index and the staging record with it.

Docs

docs/cli_reference.md (preloop volumes gc, the reasons, the four variables), docs/vm-substrates.md (a collection bullet and the variables), CHANGELOG.md (two bullets, plus #618's own bullet no longer claims nothing collects leftovers).

Review fixes

  • A volume this engine published still counts as live only while AgentENV reports it ready (or inside the grace period). A published volume that goes failed is now reclaimed as unusable once it is past the grace period, and the next start populates that tree again instead of failing on it. populate_volume records the final volume's creation time before checking its readiness, so an attempt that ends in a failed volume still has an age instead of looking ageless for ever.
  • Deletions are re-checked against the facts that can have changed since the plan: a staging record this pass did not sample (a populate that started while the listing ran), this process's in-flight stages, and the provider's own published names. What that saves is reported as left instead of silently surviving.
  • The staging registry's read-modify-write is serialized — a process-local mutex plus an exclusive lock on a sibling agentenv-volume-staging.json.lock (no new dependency) — and each write gets its own temporary file name, so overlapping writers no longer drop each other's record.
  • preloop volumes gc exits nonzero when a planned deletion failed, as prune does.
  • The periodic collector starts with the engine (VmProvider::start_maintenance from RunnerPool::run) instead of with the first volume it needs, so an engine that only resumes sandboxes still collects.