fix(agentenv): run guest-privileged steps through sudo -n; no runner dns on agentenv

Closed
BN
7d ago
+412-107

Why

The AgentENV backend could not complete a golden build against the official runner image. Four gates failed in order, all from bake runs on 2026-10-15 (logs under /var/tmp/aenv-bake-root):

  1. Golden build, DNS: Error: invalid VM configuration: AgentENV exposes no per-sandbox resolver setting; \dns` cannot be honoured. The build-golden spec always sets dns: Some(runner_dns())`, and the AgentENV provider refuses any per-sandbox resolver. The runner pool spec has the same pattern, so the engine would fail at job start.
  2. Disk preflight: golden build needs 84.0 GiB free … only 72.1 GiB is available. An operator disk condition rather than a code defect. Cleared by freeing scratch space on the host; no preflight was skipped.
  3. Guest staging, privilege: smolvm prepare_guest_dirs failed with exit code 1: mkdir: cannot create directory '/run/preloop-vm-staging': Permission denied. The runner image declares USER runner (uid 1001). aenv exec runs as that user and has no user flag, while the provider's own steps need root.
  4. Pack after stop: smolvm pack failed with exit code 1: Error: HTTP 400: sandbox cannot be snapshotted from paused state. The orchestrator stops a golden and then packs it. On AgentENV, stop is a pause, and only a running sandbox can be snapshotted.

What changes

  • DNS (b096e367). The runner's default resolver is requested only for the smolvm backend (runner_dns_for), in both the build-golden spec and the runner pool spec. An explicit dns is still refused by the AgentENV provider.
  • Privileged provider steps (38bf77d3). prepare_guest_dirs, volume extraction and cleanup, and the secret-env read and removal run through the shared root-or-sudo wrapper, preloop_vm::guest_privilege, moved from the orchestrator unchanged (a7a86807). A root exec runs them directly; the image user runs them under sudo -n. A missing passwordless sudo fails the step with an error that names the prerequisite.
  • Uploads stay unwrapped: envd writes them as root and chowns them to the exec user. Payloads keep the exec user, as before. Extracted volume trees are chowned to the invoking user, matching their previous ownership.
  • Resume before snapshot (e4cacbe8). pack, and the snapshot a fork takes of its base, resume a non-running sandbox and wait for the guest first, the same way start does for a paused machine.
  • A doc comment for as_runner_user that had been merged onto run_as_root_or_sudo is reattached to its function (a7a86807).

Evidence

  • Throwaway sandbox from the pinned runner image (ghcr.io/preloopdev/runner-images:ubuntu24-runner-large-latest@sha256:4f7e4be4…; amd64 manifest sha256:bae6ef…), run as the default exec user: uid=1001(runner) gid=1001(runner), SUDO_OK, MKDIR_OK.
  • Upload ownership on the same image: an upload into a root-owned staging directory succeeded (envd is root), the file came out owned by runner, and rm of it from the root-owned directory returned Permission denied for the exec user. That is why removal is privileged.
  • cargo test -p preloop-vm: 41 library, 22 agentenv provider contract and 17 provider tests pass.
  • cargo clippy -p preloop-vm --all-targets -- -D warnings: clean. cargo fmt --check clean for the touched crates.
  • cargo test -p preloop-orchestrator sudo_wrapper and runner_user_wrapper_survives_a_sudoless_bare_base: pass.
  • cargo test -p preloop-cli runner_dns_is_requested_only_where_the_backend_can_honour_it: passes.

Commits

  • a7a86807 refactor(guest): share the root-or-sudo guest wrapper with preloop-vm
  • 38bf77d3 fix(agentenv): run root-only guest steps through sudo -n
  • b096e367 fix(cli): do not request a runner dns on the agentenv backend
  • e4cacbe8 fix(agentenv): resume a paused machine before snapshotting it

Scope

The smolvm backend is unchanged: its provider is not touched, and it still receives the runner's default resolver. The wrapper applies only to the AgentENV provider's own steps, and the resume applies only where AgentENV would otherwise refuse a snapshot.

Bake status (2026-10-15): blocked at publish, scratch on /tmp

With e4cacbe8 the AgentENV golden build gets through guest preparation, the DNS gate, and the pack-from-pause gate. It then fails at publish. The snapshot is captured, and publish-time compression ([snapshot.publish_compression], enabled) writes its zfile scratch under temp_dir(). The AgentENV server has no TMPDIR and no PrivateTmp, so that scratch is /tmp, a 16 GiB tmpfs with about 3.1 GiB free. The 7.64 GiB raw layer ran out with No space left on device (os error 28) during the merge. No alias was published, and nothing from the attempt is left running.

The fix is a server-side TMPDIR on the root filesystem, pending approval; it is not a preloop code change. Preloop's disk preflight checks only the builder volume, so it would not catch this scratch limit.

Summary by CodeRabbit

  • Bug Fixes
    • DNS settings now follow the selected VM backend: SmolVM uses its configured or default resolver, while AgentENV does not request one.
    • AgentENV setup and volume extraction handle guest permissions more reliably. Secret files are cleaned up after failed runs when possible; privileged operations require passwordless sudo when the guest is not running as root.
  • Improvements
    • Paused AgentENV sandboxes are resumed and checked for readiness before snapshots are created.