fix(golden): give the runner the kvm group and a world-accessible /dev/kvm

Closed
BN
7d ago
+381-1

Stacked on #615 (fix/agentenv-guest-privileges). Base of this PR is that branch.

Why

Nested KVM is what the AgentENV bake's validation probe and every KVM job open. The guest's /dev/kvm stayed root:root 0600, and the runner account was not in kvm, so uid 1001 could not open the device. The probe runs through aenv exec.

Two separate problems:

  • Mode. Group membership alone does not reach a process started by aenv exec: it carries only the uid's own group (/proc/self/status Groups: 1001 1001). The node has to be world-accessible. Hosted runners ship exactly that: KERNEL=="kvm", GROUP="kvm", MODE="0666".
  • Membership. The runner was not in kvm in the golden. The cause is not established (see below), so this change reports the reason instead of hiding it.

What

Golden contract (golden_contract_steps), last two steps:

  • kvm_udev_rule_script: writes 99-preloop-kvm.rules with MODE="0666" for any guest that runs udev.
  • runner_kvm_access_script(user, "/dev/kvm"): creates group kvm if missing, hands /dev/kvm to it at mode 0666, then adds the runner with usermod -aG (falling back to gpasswd -a). It checks membership in /etc/group and warns if the runner is still absent. Every step is idempotent. A step that cannot land writes a preloop: warning: line to stderr; the contract still exits 0, so the bake continues.

Engine:

  • apply_golden_contract logs the contract's stderr as a warning when the contract succeeds. Before this, those lines were discarded.
  • Each job launch (as_runner_user provisioning) runs the same access script after the docker group. A golden that lost the membership or the grant still gives job steps the group and the node.

Not established

Why the membership was missing in the golden. Ruled out:

  • /etc/gshadow inconsistency: on Ubuntu 24.04, usermod -aG succeeds when the group exists only in /etc/group (checked in a throwaway container).
  • A later step rewriting groups: nothing after the account step removes memberships. Every later writer is append-only (usermod -aG docker in the golden and at each launch).

Remaining candidate: usermod failed in the golden with its error discarded by 2>/dev/null || true. The new warning carries that error into the engine log on the next bake.

Tests

Run in preloop-orchestrator's library suite:

  • runner_kvm_access_opens_the_device_to_every_process: mode 0666, group, device, then membership, in that order; no gpasswd unless usermod refuses.
  • runner_kvm_access_is_idempotent_across_boots
  • runner_kvm_access_falls_back_to_gpasswd_when_usermod_refuses
  • runner_kvm_membership_failure_is_reported_not_swallowed
  • runner_kvm_access_without_a_device_still_adds_membership
  • runner_kvm_access_reports_a_refused_group_and_still_sets_the_mode
  • kvm_udev_rule_grants_the_device_at_mode_0666
  • kvm_membership_is_the_last_group_write_in_the_golden_contract
  • runner_launch_adds_kvm_membership_after_docker
  • every_generated_script_parses_as_posix_sh covers both new scripts.

Notes

  • Changing the contract changes the golden fingerprint for configured images, so existing bakes rebuild. Intended.
  • The grant is applied while the golden is baked, and the per-launch step re-asserts it, so a restored snapshot keeps working even if its devtmpfs node is recreated.

Test status (cpane, cargo test --release -p preloop-orchestrator --lib)

  • Single-threaded: 150 of 151 pass. All of this change's tests pass.
  • The one failure, lifecycle_tests::sudo_wrapper_strict_variant_preserves_failures, is host policy and predates this change: on the test host sudo -n true is allowed while sudo -n sh -c 'exit 0' is refused, and the test assumes the two agree. The wrapper is unchanged here.
  • In the parallel run, environment::tests::fingerprint_tracks_the_official_golden_source can fail. It passes alone and single-threaded. Five tests set PRELOOP_GOLDEN_URL without serialising against it, and the official source reads that variable. The same five calls exist on the base commit, so this predates the change.
  • End-to-end: the generated golden contract was run in a clean ubuntu:24.04 container as root (dash). Exit 0; runner is in kvm in /etc/group; /dev/kvm is root:kvm mode 0666; the udev rule is written. The container has no /etc/sudoers.d, so the contract prints one harmless line for the sudoers drop-in; the golden image ships sudo.

Devin Review

Summary by cubic

Makes /dev/kvm accessible to the runner so KVM jobs and the bake's validation probe can open the device.

The guest's /dev/kvm was root:root 0600 and the runner was not in the kvm group, so uid 1001 could not open it. The golden contract now hands the node to group kvm at mode 0666 (as hosted runners ship it), adds the runner to kvm via usermod -aG with a gpasswd fallback, and writes a udev rule for guests that run udev. Each job launch re-asserts the group and the grant after the docker group.

  • Membership alone is insufficient: aenv exec gives its processes only the uid's own group, so the node must be world-accessible.
  • The contract no longer swallows step failures; its stderr is logged as a warning, and per-step warnings identify what could not land.
  • The contract still exits 0 when a KVM step fails, so the bake proceeds.

Existing bakes will rebuild because the golden contract changed.