refactor(sandbox): separate machine lifecycle from the CI control plane

Merged
BN
2d ago
+14770-10767

Purpose

Separate machine lifecycle from CI scheduling. Add a control-plane-independent Sandbox Service and adapt the existing runner pool to use it. This is a draft for continued sandbox work, not a claim that GitHub-hosted scheduling or official-runner pool support is complete.

Changes

  • Replace preloop-vm with preloop-sandbox: templates, pools, leases, exec, file transfer, snapshots, provider agents, and reconciliation.
  • Add the SandboxMachines adapter and route existing runner/golden flows through it while retaining current-main pool supervision and lifecycle behavior.
  • Add the standalone preloop sandbox CLI.
  • Replay only the sandbox change from the original worktree onto current public main; exclude unrelated private-main history.
  • Port to current workspace dependencies and update public docs.
  • Public driver implementation is SmolVM; non-public backend implementations are maintained only in a separate private PR.

Review corrections

  • Secret-bearing exec scripts must be archived with mode 0600.
  • Recover persisted snapshot golden templates when a new adapter starts, preserving restart adoption.

Verification

  • cargo check --locked -p preloop-sandbox -p preloop-orchestrator -p preloop-cli passed after lockfile reconciliation.
  • Final targeted tests: 73 sandbox tests and 19 adapter/pool lifecycle tests passed, including the two review regressions.
  • cargo fmt --all -- --check, clippy with --all-targets -- -D warnings for CLI/sandbox/orchestrator, and the CLI build passed.
  • Actual standalone CLI/API smoke: provider listing, template and pool creation, and SQLite restart persistence passed in a temporary home with no local VM provider. Confirmed this did not start the CI engine or download an image.
  • Hardware-specific driver behavior is not established by the fake-driver tests.

Remaining sandbox review work before merge

  • Enforce or explicitly reject incompatible fork overrides (network, mounts, sockets, machine shape).
  • Make remote exec completion reliable across failed report delivery.
  • Scope provider reconciliation and host cleanup to the owning service; avoid cross-instance VM deletion.
  • Review local API authentication/Origin handling, pause/resume state transitions, transfer expiry, and durability guarantees.

Not included

GitHub runner-management API client, JIT intake, official runner lifecycle adapter, GitHub queue/demand integration, or a standalone preloop runners pool service. These compose above the sandbox API, rather than putting runner concepts in the machine service.

Summary by cubic

Separates machine lifecycle from CI scheduling by replacing preloop-vm with a control-plane-independent Sandbox Service (preloop-sandbox). The runner pool now gets machines through the service, and a standalone preloop sandbox CLI drives it. This is a draft for continued sandbox work; GitHub runner management and the official runner pool adapter are not included.

  • Adds a Connect API (proto/preloop/sandbox/v1) for templates, pools, leases, exec, snapshots, and file transfer over HTTP/1.1; state persists to SQLite or Postgres and survives restart.
  • Adds SandboxMachines, the adapter that maps the runner pool's machine vocabulary onto the Sandboxes API.
  • Adds the standalone preloop sandbox CLI for templates, pools, acquire, exec, snapshots, and agent registration; remote host agents long-poll the service for work.
  • Removes preloop-vm, PRELOOP_VM_BACKEND, AgentENV and preloop volumes; substrates are now drivers attached to host agents, with only the SmolVM driver in the public build, driven via PRELOOP_SANDBOX_POOL_DRIVER.
  • Secret-bearing exec scripts are archived with mode 0600, and snapshot golden templates are recovered on adapter restart.
  • Waits on the smolvm child and host tools run through the shared preloop_observability::deadline helpers, bounded (probes 30s, control 5min, boot/transfer 30min, packs 60min, guest command 40min) with VmError::TimedOut; truncated captures are refused and a wedged sweep fails instead of hanging.
  • Debug attach no longer resumes a suspended guest before writing the ACTIVE marker.
  • Makes the lease-expiry and provider-loss tests deterministic instead of timing-dependent, and keeps the conformance harness provider inside its liveness window.
  • Adds docs/sandbox-service.md covering the service, its configuration, and the CLI.

Summary by CodeRabbit

  • New Features
    • Added the Sandbox Service and preloop sandbox commands for managing sandboxes, templates, pools, and providers.
    • The CLI and CI VM pool now use the Sandbox Service with SmolVM as the supported driver.
  • Changes
    • AgentENV support and the preloop volumes commands are no longer available.
  • Bug Fixes
    • Golden-release calls now use shared circuit-breaker handling; snapshot captures are limited to 90 minutes, with cleanup on expiry.
  • Documentation
    • Added a guide to the Sandbox Service, its configuration, and CLI.