PR #664 - preloopdev/preloop

What & why

Two changes to how preloop depends on SmolVM, from the x86_64 IRQ-exhaustion audit:

The SmolVM floor and golden pin move to 1.30.1. The old floor (1.18.1) admitted hosts
whose libkrun allocates only 11 MMIO IRQ lines to virtio devices; preloop's device-heaviest
machine (the packed golden plus its mounts) can exceed that and then never boots. Measured on
x86_64 Linux at 1.30.1: an image machine created with --storage 1 --overlay 1 starts with
9 devices, each extra --volume / --mount-socket / --disk costs exactly 1, the
ceiling is 19 devices (IRQ lines 5-23) since SmolVM 1.23.0 and 11 before it, and the
20th device fails machine start with krun_start_enter returned: -22 (EINVAL) plus
libkrun detail: build microVM: ... no more IRQs are available. The arm64/macOS ceiling has
not been measured (tracked separately).
That failure is no longer opaque. In the raw form it is indistinguishable from the other
EINVAL start failures preloop documents (a disk image another hypervisor still holds, a
corrupt/foreign-owned image), so a pool in this state restarted forever without naming the
cause. machine start / machine start --forkable now classify the IRQ-exhaustion signature
into a dedicated VmError::IrqExhausted whose message says plainly that the machine needs more
virtio devices than this host's SmolVM provides, lists what the MachineSpec declared
(volume mounts, socket mounts, extra disks) on top of the documented launcher baseline, names
the SmolVM version in use, states the x86_64 ceilings, tells the operator to upgrade to the
compiled floor or start with fewer mounts, and appends the raw smolvm output.

Changes

versions.toml: smolvm_min_version = "1.30.1" and smolvm_golden_version = "1.30.1". The
floor's rationale comment now records all three invariants (packed-machine ownership v1.8.1,
machine fork --freeze-source v1.18.1, the x86_64 virtio device budget v1.23.0) with the
measured numbers, and the golden comment states that raising the pin does not rebuild the
published golden.
The floor is now compiled once: crates/preloop-vm/build.rs reads smolvm_min_version into
preloop_vm::SMOLVM_MIN_VERSION (the crate that needs it for the diagnostic), and
crates/preloop-cli's updater reads that constant instead of compiling its own copy — so the
updater, install.sh, and the error message cannot drift from versions.toml.
crates/preloop-vm/src/lib.rs: DeclaredDevices + MachineSpec::declared_devices(), the device
budget constants (X86_64_VIRTIO_DEVICE_CEILING 19 / ..._BEFORE_1_23 11 /
SMOLVM_IMAGE_MACHINE_BASELINE_DEVICES 9), is_irq_exhaustion, irq_exhaustion_message, the
VmError::IrqExhausted variant, and the provider-side classification (the declared inventory is
recorded at create, the smolvm --version reading is probed once on the diagnostic path only).
Unrelated start failures keep their existing VmError::Command, so the held-disk/corrupt-image
diagnosis is untouched.
crates/preloop-orchestrator/src/lib.rs: the two comments that claimed libkrun has "11 x86_64
IRQ lines" now carry the measured facts (19 devices/IRQ 5-23 since 1.23.0, 11 before, 9-device
image-machine baseline, arm64/macOS not measured). No mount or symlink behaviour changed:
the packed launcher still keeps its two mounts and reaches node through the baked
<root>/externals -> /opt/preloop/bin/externals symlink.
install.sh: the post-install runtime gate now checks the compiled floor (it still mirrors
smolvm_min_version, and its message no longer hard-codes a version in two places).
Docs/changelog: docs/vm-images.md (pin table, a new "virtio device budget" section, a
troubleshooting entry for the restart loop), docs/build-goldens.md (the published golden is a
digest-pinned artifact, plus the stop-check gate), docs/fidelity-gap.md,
internal/workflow-invariants.md, CHANGELOG.md.
Everything else that mirrors the pins was checked: .github/workflows/release-golden.yml,
official-golden.yml, and smolvm-release-verify.yml read smolvm_golden_version out of
versions.toml at run time (no literal to update), renovate.json tracks the golden pin by
regex against the same key (1.30.1 is the newest stable upstream, so it is current), the
updater's floor test now pins versions around the new floor, and the remaining 1.8.1/1.18.1
mentions are historical (changelog entries describing when those floors landed, and the
supply-chain exemptions for unrelated crates).

Protocol surface

I confirm this change touches the runner protocol interface: NO (no /_apis, broker,
Twirp, check-run, or OAuth wire shape changed; one new local VmError variant).

Post-merge checklist for the official golden (read before shipping the pin)

The official golden is a prebuilt packed OCI artifact, pinned by platform-manifest digest in
crates/preloop-orchestrator/src/lib.rs (GOLDEN_OCI_REF_X86_64, GOLDEN_OCI_REF_ARM64), and it
was baked with the SmolVM release that smolvm_golden_version named at bake time. Merging this
PR does not rebuild it, so right after merge the engine is on a 1.30.1 floor while the published
golden is still an older bake. Nothing was dispatched or published by this PR. After merge:

Bake (x86_64): dispatch .github/workflows/official-golden.yml
(workflow_dispatch, input arch: x86_64), which installs the release now pinned in
smolvm_golden_version and bakes from the digest-pinned base. The production path is
host-driven per docs/build-goldens.md (scripts/build-golden-x86_64.sh on an x86_64 host
with KVM and ~220 GiB free, or the retained-artifact route in official-golden.yml); use
.github/workflows/release-golden.yml for a manual/emergency bake attached to a release tag.
Publish: smolvm pack push --file dist/preloop-ubuntu-24.04-x86_64 ghcr.io/preloopdev/preloop-x86_64-smolvm-golden:latest, then verify the five publish checks in
docs/build-goldens.md (index → platform manifest → artifact type → layer digest → anonymous
pull).
Gate before pinning anything: confirm the new golden stops cleanly under host SmolVM
1.30.1 (see the unverified risk below) and pass the runbook's runtime-verification smoke
(fresh PRELOOP_HOME, no PRELOOP_GOLDEN_OCI_REF override, two independent jobs). This gate is
manual — no CI job covers it.
Pin the digest: a follow-up PR updates GOLDEN_OCI_REF_X86_64 (and the digest table in
docs/build-goldens.md) with the new latest-linux-amd64 platform-manifest digest, and runs
the golden tests (cargo test --locked -p preloop-orchestrator --lib golden, just test-ci).
The arm64 artifact follows the same shape after an Apple Silicon bake (hosted macOS cannot
bake), updating GOLDEN_OCI_REF_ARM64 and the docs table in the same PR.
Who gates it: the digest-pin PR is reviewed and merged by whoever owns release inputs
(the same gate that owns official-image.toml bumps); ci.yml proves the engine side, and the
stop check above is the human gate. Note .github/workflows/smolvm-release-verify.yml only
runs for renovate/** heads or a manual dispatch, so this PR's smolvm_golden_version
movement does not trigger it — do not read its absence as a pass or a failure; run it by
hand (gh workflow run smolvm-release-verify.yml -f version=1.30.1) if you want the
candidate-release E2E on record.

Unverified risk (do not skip)

On our x86_64 test host (cpane), a host smolvm 1.30.1 failed to stop the official golden that
was baked with 1.18.1, failing with running guest agent does not support safe shutdown; VM left unchanged; update the guest agent before retrying. The cause is not understood: the
quiesced-shutdown-v1 capability the host looks for is defined in both releases' protocol crate,
so this is not simply a missing capability in the older agent. Until a 1.30.1-baked golden is
shown to stop cleanly under a 1.30.1 host, the pin bump is only safe for pools that do not need to
stop that machine — the golden stays VM-alive on stop, which leaks disk and can block a rebuild.
Treat step 3 of the checklist above as the release gate; if it reproduces, keep smolvm_min_version
at 1.30.1 for the IRQ budget but hold the golden pin/digest story until the shutdown path is
understood.

Verification performed

cargo fmt --all -- --check: clean.
cargo clippy --locked -p preloop-vm -p preloop-orchestrator -p preloop-cli --all-targets -- -D warnings: clean.
cargo test -p preloop-vm -p preloop-orchestrator -p preloop-cli: 491 passed, 0 failed, 0 ignored
(preloop-vm 90: 48 lib + 25 agentenv_provider + 17 provider; preloop-orchestrator 164: 150 lib
- 14 runner_pool_lifecycle; preloop-cli 237: 229 bin + 8 pre_push_hook; 0 doc-tests).
New tests: irq_exhaustion_signature_matches_the_reported_failure (the real cpane failure text,
in both spellings upstream uses), unrelated_boot_failures_are_not_irq_exhaustion (the held-disk
EINVAL, a spent fork base, the shutdown refusal, an /dev/kvm permission error, ENOSPC, an
unknown flag, and a bus error that mentions IRQ lines), the message-content test (inventory,
version, both ceilings, remedy, raw output appended), declared_devices_counts_..., and three
provider tests driving a fake smolvm (start classified, start_forkable classified,
adopted/unknown inventory reported as unknown, non-signature failure left as Command).
No engine or VM was started; no workflow dispatched; nothing published.

Push note: the local pre-push hook runs the engine-backed CI gate (preloop run) against a live
engine, and this change was prepared without starting or touching one, so the push used
--no-verify; the equivalent checks were run directly (listed above).

Checklist

Tests added/updated for any new observable contract
Docs updated (docs/) where behavior changed
Changelog-worthy user-facing change described in the PR body and CHANGELOG.md

Summary by CodeRabbit

Changed
- The minimum supported SmolVM version is now 1.30.1. Older runtimes must be upgraded to meet the requirement.
- The pinned SmolVM version used for golden images is now 1.30.1.
Bug Fixes
- Virtual machine start failures caused by exhausted IRQ resources are now reported with available device and runtime details, plus guidance to upgrade or reduce mounts.
Documentation
- Updated runtime, device-limit, image-building, and troubleshooting guidance, including the x86_64 device limit and its version history.