PR #618 - preloopdev/preloop

Summary

Publish AgentENV read-only host trees as content-addressed volumes (preloop-runner-<sha256 tree, 12 hex>) and attach them with aenv start --volume.
Populate once through an exclusive staging volume, verify the archive digest in-guest, freeze a read-only fork, and reuse it across starts with per-process memoization that is revalidated on every use.
Keep AgentENV fork bases volume-free: AgentENV 0.2.1 rejects snapshots of sandboxes with read-only volumes. fork_with_volumes attaches the current bundle to each clone instead.
The complete bundle carries Node externals; AgentENV uses one mount at /opt/preloop/bin and links /home/runner/externals into it. preloop volumes list|prune manages lifecycle cleanup.

Base and the dependency split

This branch is based on chore/preloop-vm-hex-sha2 (#665, chore(deps): add hex and sha2 to preloop-vm), not on main.

The volume work needs sha2 and hex, which changes Cargo.lock. The repository requires a lockfile change to be a dedicated dependency PR, so that change is now #665: crates/preloop-vm/Cargo.toml plus the two lockfile edges inside the existing preloop-vm entry (no new packages, no version changes). It passes the supply-chain guard as a dedicated dependency PR — no dependency-migration label, which this PR never relies on.

Merge order: #665 first, then this PR retargets to main and rebases. git diff chore/preloop-vm-hex-sha2...HEAD -- Cargo.lock crates/preloop-vm/Cargo.toml is empty: this PR no longer touches either file. Its diff against its base therefore contains no policy file at all.

Verified locally with the new guard script (PR #666, supply-chain/guard-policy-files.sh): the dependency PR (main → chore/preloop-vm-hex-sha2) reports Dependency/policy PR detected: OK for Cargo.lock + crates/preloop-vm/Cargo.toml, and this branch (chore/preloop-vm-hex-sha2 → this head) reports Policy file guard: OK (no policy or manifest files modified).

Rebased onto `main`

The branch was rebuilt from main @ fd08d18 (the merge of #620) by cherry-picking only this PR's own commits. The other four commits that used to be on this branch (the guest-wrapper refactor, sudo -n, the DNS fix, resume-before-snapshot) are already on main in rebased form via #620, so they were dropped rather than replayed.

Conflicts, and how each was resolved:

crates/preloop-vm/Cargo.toml / Cargo.lock — resolved by the split: the dependency PR owns them, this branch's commits no longer touch them.
CHANGELOG.md — both sides added entries under ## [Unreleased]. Kept main's structure and appended this PR's entry to main's ### Changed.
crates/preloop-vm/src/agentenv.rs — merged textually clean, then checked hunk by hunk. #620's snapshot behaviour is intact and this PR's volume path replaces the old materialization path:
- resume_if_paused and snapshot_running are unchanged from main, including the re-pause on success and on error, and both call sites (pack and the fork path) still go through snapshot_running.
- The old per-start copy path (materialize_volumes, ensure_volumes, volumes_materialized, tar_directory) is gone, replaced by read_only_mounts/ensure_read_only_volume/populate_volume.
- prepare_guest_dirs stays, reduced to the secret directory it still owns; start keeps main's "record the id before anything else can fail" ordering and now passes record.spec.volumes into start_sandbox_supervised.
crates/preloop-vm/tests/agentenv_provider.rs — merged clean. Every test main gained in #620 is kept and none needed a signature change: guest_scratch_directories_are_prepared_through_the_root_or_sudo_wrapper, a_guest_without_passwordless_sudo_fails_start_with_the_prerequisite_named, an_explicit_dns_is_refused_before_anything_runs_and_an_absent_one_starts, pack_resumes_a_paused_machine_before_snapshotting, pack_pauses_the_sandbox_again_after_snapshotting_it, pack_pauses_the_sandbox_again_when_the_snapshot_fails, fork_pauses_the_golden_again_after_snapshotting_it.
crates/preloop-orchestrator/src/lib.rs, crates/preloop-vm/src/lib.rs — merged clean; #620's code is untouched and the volume capability/fork changes are additive.

Two tests are removed rather than adapted, because the code they covered no longer exists: host_volumes_are_streamed_in_as_a_tar_and_read_only_mounts_lose_write_bits and a_clone_inherits_materialized_volumes_instead_of_re_uploading_them. They asserted the old copy-into-the-guest design that this PR replaces.

Review fixes

Stale volumes (devin, agentenv.rs). A remembered volume is no longer trusted: each start re-derives a cheap tree fingerprint (a metadata walk — names, types, modes, sizes, mtimes, link targets, plus the version pins — not an archive) and re-checks that the volume still exists before mounting it. An in-place bundle update therefore publishes a new volume, and a volume deleted under a running engine is populated again instead of being mounted dead.
One lock for every tree (devin). The single mutex around archiving and populating is now one lock per host tree: different trees populate concurrently, repeated starts of one tree still populate once.
Symlinks that escape the volume (devin, security). A symlink whose target is absolute, or whose relative target climbs out of the tree with .., fails the archive with an error naming the link and its target. Every link in the tree is checked, so a chain cannot escape through an unchecked link either. The six links in the real bundle (externals/node{20,24}/bin/{npm,npx,corepack} -> ../lib/node_modules/...) are relative and in-tree, so nothing legitimate is refused.
Set-user-ID, set-group-ID and sticky bits. mode_of masked 0o7777; it now masks to 0o777, for directories as well as files, so a host bundle cannot hand a guest a setuid binary or a sticky directory. A tree that had those bits publishes a new volume (new name), which prune reclaims. AgentENV decides the volume's mount flags, so nosuid/nodev are not something this backend can set or verify.
Manifest inside the volume. The archive carries a synthetic .preloop-bundle.json at the tree root — never written to the host directory — with sorted keys and nothing time- or host-dependent: {"bundle_bytes":…,"bundle_files":…,"node_externals":{"node20":…,"node24":…},"runner_version":…,"schema":1}. The versions come from versions.toml through preloop_orchestrator::runner_bundle_manifest() (build.rs compiles the pins into the CLI and the orchestrator, not into preloop-vm, so the provider is handed the manifest); an unknown pin is recorded as unknown instead of failing. It is part of the digest, so a version change publishes a new volume.
Local volume index. When a populate succeeds the engine writes <preloop home>/agentenv-runner-volumes.json (atomic, 0600, best-effort) mapping the volume name to the manifest, the host tree, and the creation time — the creation time lives only there, never in the tree. preloop volumes list prints runner/Node versions from that index and marks a volume with no entry unknown (pre-manifest or created elsewhere), because AgentENV cannot be asked what is inside a volume without mounting it.
Default prune kept the wrong bundle (devin, codex P2). The default keep list is now the same selection the pool makes: discovered_runner_bundle() (the same discovery local_runner_pool_config uses, PRELOOP_RUNNER_BUNDLE or the installed/development bundle beside the binary) passed through preloop_orchestrator::effective_runner_bundle_path() (including the engine-owned mirror), plus the standalone externals tree only when it exists. A missing externals tree no longer aborts a prune.
Dry run misreported mounted volumes (devin). A dry run now reports candidate for each volume a real run would attempt to delete, and claims nothing was deleted. It cannot report in use for a specific volume: AgentENV reveals a reservation only when a delete is attempted (HTTP 409: volume is reserved by sandbox …), and no read-only API exposes it (the CLI's volume record is {volumeID, name, mode, sizeMB, status}, and aenv ls --output json drops the sandbox's volumeMounts). The real run still classifies from the server's own refusal, so its in use list is authoritative.
Pruning versus a running engine (coderabbit). The doc claim that a deleted volume is simply repopulated is now true rather than aspirational, and is stated where it belongs: with revalidation, deleting a live volume costs a repopulate, not a failure; the unsafe window is a volume deleted while it is being populated (between its create and its first mount). delete_volumes(names) was extracted so a caller that plans a set (the automatic collection pass being built on top of this branch) can delete exactly what it classified.
Lockfile threads (devin, codex P1). Answered by the split above: #618 no longer touches Cargo.lock; the two edges live in #665, and no label is involved.
Rename. preloop runner-volumes is now preloop volumes (list|prune), crates/preloop-cli/src/runner_volumes.rs → volumes.rs, and the provider API follows: list_volumes/prune_volumes. list gained a KIND column (runner-bundle, or runner-bundle (staging) for a populate leftover). The volume name prefix stays preloop-runner- for runner-bundle volumes. The command is unreleased, so this is a clean cutover: no alias, no shim.

Evidence

Experiments on cpane used only rvtest-* resources, one sandbox at a time, and were cleaned up. No engine/config/golden/zot changes were made.

Bundle: 422,129,834 bytes, 11,157 entries; host deterministic tar: 431,400,960 bytes in 2.26 s.
Exclusive populate: volume create 1.47 s, sandbox start 3.06 s, 431 MB upload 0.95 s, unpack 1.01 s, commit on sandbox delete 54.6–58.4 s, read-only fork 1.3 s.
Host and guest manifests match (paths/types/modes/symlink targets/content hashes).
Read-only mount: uid 1001, ext4 ro,relatime,norecovery, no noexec; runner and Node 20/24 execute. User and root writes fail with EROFS; root remount fails.
Two simultaneous readers mount successfully. Delete while mounted is refused with HTTP 409: volume is reserved by sandbox ....
Pause/resume with a read-only mount works.
Snapshotting a sandbox with a read-only mount fails in AgentENV 0.2.1 (HTTP 500 ... volume ... must enter uploading state). A plain snapshot with no volume, followed by clone starts with --volume, works; two such clones run the runner and Node successfully.
A single bundle mount plus rm -rf /home/runner/externals && ln -s /opt/preloop/bin/externals /home/runner/externals works end to end for runner and Node binaries. A second direct mount at /home/runner/externals fails in envd; the implementation therefore uses the single-volume layout.
After deleting the exclusive source, a fresh ro mount produced identical
full manifests: host 84e1230621aea3cfa64fafb88853963fae5ba896f021df2554ecbbb068499993
/ b2155bc3ffd70e6ab8a45cbb17efcf9c04ee01a3ebe0aa05aca05b67312b4d5a;
guest produced the same two digests. Runner 0.2.0 and Node 20/24 executed
from that post-delete mount.
An adopted-alias smoke clone ran as uid 1001, linked
/home/runner/externals -> /opt/preloop/bin/externals, executed Node 20/24
and runner 0.2.0, and matched runner SHA256
1151129f0403b355b632804e54ac2e31b4527a2d8e03a4f681415eb57c906dda.
The six symlinks in the real bundle (~/preloop-aenv/bundle) are all relative
and in-tree, which is what the archive's symlink check expects.

Verification (on the rebased tree)

Run in clean worktrees on this machine with the repo's kache build cache. Test commands were run with --locked (the gate commands plus a lockfile check):

Gate	This head	Previous head	`main` today
`cargo fmt --all -- --check`	clean	clean	—
`cargo clippy --locked -p preloop-vm -p preloop-cli -p preloop-orchestrator --all-targets -- -D warnings`	clean (exit 0)	clean	—
`cargo test -p preloop-vm`	107 passed	96	83
`cargo test -p preloop-cli volumes`	4 passed	2	0 (no such command)
`cargo test -p preloop-orchestrator`	164 passed	164	164

Counts: preloop-vm = 53 lib + 37 tests/agentenv_provider.rs + 17 tests/provider.rs + 0 doc-tests. preloop-orchestrator = 150 + 14 + 0 doc-tests. No failures, no ignored tests. cargo vet --locked on the dependency PR: Vetting Succeeded (17 fully audited, 2 partially audited, 467 exempted).

Where the growth comes from: +11 in preloop-vm (six volume unit tests — manifest determinism, a changed pin, an unknown pin, escaping symlinks, masked mode bits, a plain tree — and five provider tests — in-place update, pruned volume, index round-trip, dry-run candidates versus a mounted-volume refusal, concurrent populates of different trees), and +2 in the CLI (volumes) for the keep list and the list rendering. The orchestrator suite is unchanged: the test-name set is identical to main's.

CI notes

zizmor (impostor-commit on dtolnay/rust-toolchain@686976e1…) is a stale run, not this branch. The failing check flagged that pin at ci.yml lines 48/152/296/420, which is exactly the old base's file (fix/agentenv-guest-privileges, and main before 9d67035); this head's ci.yml pins b725eda5… at 48/153/299/423 and this PR does not touch .github/workflows at all. The run's own step table shows it checked out the old toolchain pin, i.e. the pre-retarget merge ref. uvx zizmor .github/workflows/ (v1.31.1) on this head reports "No findings to report". A fresh run on the new head is what settles it.
rust shard 4 of 4: tests::kill_vm_processes_in_kills_only_the_named_machines_hypervisor. Not this branch's regression: this PR's diff to crates/preloop-vm/src/lib.rs is capabilities plus fork_with_volumes and does not touch kill_vm_processes_in or that test, and the failing run also used the stale checkout above. Locally the test passes 20/20 in a loop and cargo nextest run -p preloop-vm --partition hash:4/4 passes (nextest reports it LEAKy — the test's fake _boot-vm shells leave their sleep 60 children behind — which is a nextest annotation, not a failure). It scans ps for _boot-vm processes by machine data dir, so on a shared runner it is exposed to other jobs' processes; the assertion to watch is killed == 1.

Operator switch-over

Do not deploy this PR without reviewing the AgentENV service's existing binary/unit. Once #665 and this PR are merged, keep PRELOOP_VM_BACKEND=agentenv and switch PRELOOP_RUNNER_BUNDLE to the combined bundle before replacing the engine binary:

cpane's engine currently points PRELOOP_RUNNER_BUNDLE at /home/bnjoroge/preloop-guest-privs-release3/bundle (runner only).
The combined runner bundle — the runner plus the Node externals it now carries into /opt/preloop/bin — sits in ~/preloop-aenv/bundle. Point PRELOOP_RUNNER_BUNDLE at that tree, then replace the engine binary and restart the engine so the per-process volume memo is rebuilt.

The first bundle version populates its volume once; later starts mount it by name. A changed bundle tree — including a changed version pin, because the manifest is part of the digest — is a new volume, so the old one is reclaimed by preloop volumes prune, not by a golden rebuild. Because every existing tree's digest changes when the manifest lands, the volumes published by earlier builds become unreferenced and are reclaimed by the next prune. Run prune while no runner is being provisioned; it is otherwise safe while the engine runs, because a remembered volume is revalidated before reuse and repopulated if it is gone. This PR does not deploy or modify any live engine/config.

Not verified

No real job has been run on a clone with the read-only bundle volume. The evidence above shows the runner and Node executing from a read-only mount, a pause/resume, a mounted-delete refusal, and manifest identity — not a workflow job (registration, a step, artifact/cache I/O) completing on an AgentENV clone whose /opt/preloop/bin is an ro volume. Treat that as the merge gate before deploying.
The design assumes the runner never writes under /opt/preloop/bin. That is already how production SmolVM mounts the same host tree: crates/preloop-orchestrator/src/lib.rs:107-109 mounts effective_runner_bundle(config) at /opt/preloop/bin with read_only: true and the runner is executed from there (crates/preloop-orchestrator/src/lib.rs:6971), so a read-only bundle directory is not a new assumption for the runner. It is still an assumption for this mount path: AgentENV enforces it with a block device instead of virtiofs. A grep over non-test CLI and runner code shows current_exe() only ever locates a binary to exec, never a directory to write into — but that is static reading, not a run.
A supporting local experiment on the SmolVM side (not AgentENV) ran the real bundled binaries — preloop-runner (run --once, configure --no-externals) and Node 20/24 — from the same ~/preloop-aenv/bundle tree mounted read-only both as --disk img:ro and as --volume <dir>:/opt/preloop/bin:ro: no write-class syscall anywhere under /opt/preloop/bin, no EROFS from them, and a read-write copy's tree manifest stayed byte-identical. Job-level execution was not exercised there either (configure aborts early without a CA bundle), so this narrows the gap but does not close it.
The measurements in Evidence were taken before the rebase, on cpane, against AgentENV 0.2.1. Nothing in Evidence was re-run after the rebase; only the gates in Verification were.
The manifest and the volume index are covered by unit and provider tests over the fake aenv, not by a populate against a live server: no volume has been archived and mounted with a manifest, and no index has been read back from a real engine home. The index is best-effort by design (a failed write is logged), so volumes list degrades to unknown rather than failing.
The dry run's candidate output is a deliberate limitation, not a stand-in for a reservation query: AgentENV 0.2.1 has no read API for volume reservations, so a dry run cannot say in use for a specific volume. It is also not a promise: a volume a sandbox mounts will be reported in use by the real run and left alone.
preloop volumes prune is exercised by provider tests over the fake CLI, not against a live engine. Its in use classification and the prune-while-idle caveat come from the AgentENV behaviour observed during the populate experiments, not from a production prune.
Fork bases are volume-free because of an AgentENV 0.2.1 limitation (fork_bases_may_mount_read_only_volumes: false). If a later AgentENV can snapshot a sandbox with an ro volume mounted, that flag should be revisited — otherwise every future clone keeps paying the per-clone volume attach.

Base

Rebased onto main (fd08d18, the merge of #620), then rebased onto chore/preloop-vm-hex-sha2 (#665) for the lockfile split. After #665 merges this retargets to main.