PR #686 - preloopdev/preloop

Stacks on #685; retarget to main after it merges.

Closes the preloop-vm rows of the blocking-call audit (agent://BlockingCallAudit, rows #1, #2, #6) on 44d51b523. Every child wait in the provider was unbounded, and the lifecycle operations held the provider's lock across it: a smolvm/aenv that never exited parked every VM operation in the engine for good.

What was unbounded (audit row → file:line on `44d51b523`)

row	location	blocked on
#1	`crates/preloop-vm/src/lib.rs:1082-1087` `checked_with_network` child `wait()`, taken under `lifecycle_lock.write()` (`exclusive`, :1125) and `.read()` (`concurrent`, :1166); same shape in `recovery()` :896-900 and `supports_mount_socket` :954-956	`smolvm` child that never exits
#1	`crates/preloop-vm/src/agentenv.rs:293-318` `capture()`	same for `aenv`
#2	`lib.rs:1444-1446` `get_or_try_init(supports_mount_socket)` and `lib.rs:971-986` `reported_version` (`get_or_init`, `.output().await`)	wedgeable probe cached for the process's lifetime
#6	`lib.rs:2692-2722` `detach_orphaned_layer_mounts` (`/sbin/mount`, `ps ax`, `hdiutil detach`) run synchronously on an async worker inside `async fn delete` (called at `:1783`)	`hdiutil` wedged on a busy volume

Bounded in the same file and by the same mechanism, because they are the same class and sit in the same async paths: kill_vm_processes_in (ps + kill -9, lib.rs:~3053, called from delete), purge_orphaned_vms (ps + kill), and filesystem_available_bytes (df).

Deadlines

Named consts with the reasoning in their doc comments (lib.rs), one struct WaitDeadlines per provider so tests can shrink every tier:

tier	value	operations
`PROBE_DEADLINE`	30 s	`--version`, `machine create --help`, `smolvm status`/`ls`/`data-dir`, `aenv ls`/TTL keepalive, and every host tool the orphan sweeps run (`mount`, `ps`, `kill`, `hdiutil`, `df`)
`CONTROL_DEADLINE`	5 min	`stop`, `delete`, `update`, `cp`, `aenv pause`/`delete`, guest dir prep, secret upload
`BOOT_DEADLINE`	30 min	`create`, `start`, `start_forkable`, `fork`, `machine cp`, `aenv start`/`resume`/`upload`/`download`
`PACK_DEADLINE`	60 min	`pack create`, `aenv snapshot create`
`EXEC_DEADLINE`	40 min	one guest command: smolvm's own `--timeout 30m` plus a 10 min margin

On expiry the child is SIGKILLed and reaped (2 s reap grace, so even an unkillable process cannot hold the caller) and the call fails with VmError::TimedOut { operation, program, deadline } — the message names the program and the deadline, so an operator can tell a hang from a failure.

Streaming execs (exec_stream, the runner itself) are deliberately untouched: their duration is the job's.

Two correctness details beyond the timeout itself:

A timed-out capability probe is not cached. supports_mount_socket returns the timeout error (only Ok answers are stored by get_or_try_init), and reported_version moved to get_or_try_init with the same rule, so a momentary wedge cannot disable --mount-socket for the process's whole life.
Pipes are drained concurrently with the wait. A chatty child (ps ax on a busy host) fills the 64 KiB pipe buffer and blocks in write, which looks exactly like a hang; the drain is what keeps the deadline honest (unit-tested below).

Tests

crates/preloop-vm/src/lib.rs (unit):

host_tool_deadline_kills_a_wedged_tool — a tool that never exits fails at the deadline and the recorded pid is gone.
host_tool_drains_more_than_a_pipe_buffer — 360 KB of output still exits zero (no spurious hang from a full pipe).
child_deadline_kills_a_wedged_runtime — same for the async runtime child.

crates/preloop-vm/tests/provider.rs:

timed_out_create_kills_smolvm_and_releases_the_lifecycle_lock — the first create holds the write lock and never returns; the queued create must proceed once the guard drops, and the wedged smolvm must be dead.
timed_out_socket_probe_is_not_cached_as_unsupported — after a timed-out probe, the next create re-probes and succeeds instead of failing with UnsupportedSocketMount.

crates/preloop-vm/tests/agentenv_provider.rs:

timed_out_pause_is_killed_at_its_deadline.

Fail-before: on 44d51b523 the same scenario (fake smolvm whose first machine create sleeps, then two concurrent creates) does not finish within a 5 s budget — the queued create never gets the lifecycle lock. On this branch the same two creates finish in ~1 s: the first with VmError::TimedOut, the second Ok.

Gates

cargo fmt --all --check — clean.
cargo clippy --locked -p preloop-vm --all-targets -- -D warnings — clean.
cargo test -p preloop-vm — see the check list below (focused tests + the crate suite).

Deliberately left

Shared-helper swap pending. The branch is rebased onto fix/bound-shared (0b5188b9a). preloop_observability::deadline::run_command/run_command_blocking capture output unbounded, while this slice must keep with_capture_limit/ExecOutput.truncated (an existing integration test asserts truncation, and an unbounded drain of a chatty machine exec is a memory regression). BoundShared has been asked for a capped variant; when it lands, the local run_child_to_deadline/run_host_tool collapse onto it and only the deadline consts stay.
exec_stream stays unbounded by design (the job's own duration); its liveness is the caller's probe (GuestUnresponsive).
The orchestrator/CLI call sites of detach_orphaned_layer_mounts/purge_orphaned_vms are unchanged and now call an internally bounded function; moving them onto a blocking thread is BoundOrchestrator's row (coordinated, their wrapper cannot cancel a blocking closure, so the kill had to happen here).

Summary by cubic

Bounding every child wait in the smolvm/aenv VM provider so a hung runtime can no longer freeze the engine.

Previously every child wait was unbounded, and lifecycle operations held the provider's lifecycle lock while waiting, so a runtime that never exited parked every VM operation behind it. Now each wait has a deadline: on expiry the child is killed and reaped, and the call fails with VmError::TimedOut, which names the program and the deadline so an operator can distinguish a hang from a failure. Timed-out capability probes are not cached as unsupported, so a momentary wedge cannot permanently disable --mount-socket. The orphan sweeps (mount, ps, kill, hdiutil, df) are bounded the same way and moved onto blocking threads. Streaming execs are deliberately untouched: their duration is the job's. All bounded waits go through the shared preloop_observability::deadline runners; preloop-vm keeps only the deadline constants and a WaitDeadlines struct.

Deadlines

30 s for capability and registry reads, plus the orphan sweep tools.
5 min for local state changes (stop, delete, update, cp, secret upload).
30 min for build, boot, clone, and transfer (create, start, fork, machine cp).
60 min for whole-machine export (pack create, snapshot create).
40 min for one guest command, above smolvm's own 30 min --timeout.

Inventory integrity

A ps/mount capture that hit the output cap was read as the whole host, so orphan sweeps overlooked half the inventory. A truncated table now fails the sweep with VmError::Protocol for the next tick to retry; the shared runner still drains pipes past the cap so memory stays bounded. A main merge also pulled in the nested-action 403 fix and the capped tailscale wizard query (4 MiB answer limit), unrelated to this bounding work.