Cloud VM env layers: declarative environment setup with snapshot-cached layers (cmux vm env)

LA
17d ago
+2876-11

Mitchell Hashimoto's core cloud feedback: environment setup "must magically work out of the box" ("build ghostty in your sandbox" → "zig doesn't exist"). This adds the primitive that fixes it: a declarative, GH-Actions-flavored environment spec where every step's result is a cached VM snapshot layer.

A repo declares its environment in .cmux/env.yaml (steps + verify). cmux vm env build runs each step in a Cloud VM, snapshots after each success (~0.8s on Freestyle), and registers the snapshot under a chain hash of (provider, base image, steps so far). Editing step N re-runs only N and later; renaming a step re-runs nothing; a base-image rollout invalidates everything. A fully cached spec makes cmux vm env up boot a ready environment in about a second. The agent loop (documented in skills/cmux-cloud-env/SKILL.md) is: draft spec → build --json → read failingStepIndex/logTail → patch that step → rebuild; trial-and-error setup happens once, then it's declarative and cached for the whole team.

Backend: new cloud_vm_env_layers table (billing-team scoped, never shared — snapshots can contain secrets; registration re-verifies snapshot ownership via vm.snapshot.created usage events), resolveEnvLayers/recordEnvLayer/listEnvLayers Effect workflows, routes /api/vm/env/layers[/resolve], typed errors mapped in vmWorkflowErrorResponse. Freestyle-only v1 (env_provider_unsupported otherwise). Cached-layer restores flow through the existing restoreVm → createVm gates (entitlements, credits, active-VM limits), so no new billing surface.

CLI: strict YAML-subset parser + chain hasher in CLI/CMUXCLI+VMEnvSpec.swift (a TS mirror web/services/vms/envChainHash.ts shares byte-exact test vectors), orchestrator in CLI/CMUXCLI+VMEnv.swift. Long steps don't stream through one request: the CLI stages a ~90-line POSIX runner in the VM, starts each step detached via setsid (as user cmux, bash -l), and polls a status file with short execs, so a 30-minute zig build works while every network call stays under 90s (/api/vm/[id]/exec runs in a Vercel function). Exec calls retry transient provider 502s.

Tests: PG-backed layer-cache tests (deepest-layer resolve, team isolation, idempotent upsert, ownership enforcement), 401-before-workflow coverage for the new routes, shared Swift/TS chain-hash vectors, and spec-parser tests wired into cmuxTests.

Verified live against a local dev stack + real Freestyle: create (1.5s) → snapshot → register layer → resolve deepest → idempotent re-register → ownership rejection → restore into a new VM all work end to end. In-VM step execution could NOT be live-verified because Freestyle's execution plane (exec-await + WS PTY) is currently down provider-side (https://github.com/manaflow-ai/cmux/issues/7417); the ghostty acceptance demo runs once that recovers.

Deferred: layer GC/retention cron + provider snapshot deletion (first fast-follow; lastUsedAt/invalidatedAt columns are already in place), E2B/Daytona, Swift onboarding panel, streaming logs, content-based invalidation, secrets handling.

Docs: docs/cloud-env.md (user-facing spec reference), skills/cmux-cloud-env/SKILL.md (agent loop contract), web/services/vms/README.md section.

View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.

[!NOTE]
Medium Risk
Touches Cloud VM create/restore/snapshot/exec and new team-scoped layer storage; chain hashes are client-computed with within-team trust, and Freestyle-only limits blast radius but billing/VM limits still apply on restore.

Overview
Adds cmux vm env for declarative Cloud VM setup from .cmux/env.yaml: ordered steps plus verify, with each successful step snapshotted and registered as a cached layer keyed by a deterministic chain hash (provider, base image, step prefix).

CLI wires init, build, up, layers, and logs under cmux vm env. build restores the deepest cached layer, runs only remaining steps via an in-VM detached runner (short polled vm.exec calls), snapshots intermediate layers immediately, runs verify before the final snapshot/register, and emits --json reports with finalLayerRegistered. up restores the final layer only when the spec is fully cached and spec_digest matches the current file.

Backend adds cloud_vm_env_layers, Freestyle-only resolve/record/list workflows, /api/vm/env/layers and /resolve, snapshot ownership checks on registration, and depth resolution that ignores corrupt stepIndex rows. Swift/TS share chain-hash vectors; docs and an agent skill describe the build loop.

Reviewed by Cursor Bugbot for commit b10ecb28616c1b322bc876d364dc28bcf0e9ce03. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by cubic

Adds Cloud VM env layers with a declarative .cmux/env.yaml that builds environments in a VM and snapshots each step as a cached layer for fast rebuilds and near‑instant up. up requires a passing build and a matching spec digest; the final snapshot and registration happen only after verify passes, and verify runs before that snapshot so its effects are in the cached layer.

  • New Features

    • Declarative spec at .cmux/env.yaml with steps and verify; deterministic chain hash computed by the CLI and mirrored in web/services/vms/envChainHash.ts.
    • cmux vm env build snapshots after each step and registers the final layer only after verify passes; fully cached builds refresh the final layer with the current spec digest; cmux vm env up restores only when the registered digest matches the spec.
    • Long steps run via an in‑VM detached runner with retries; per‑step script staging clears runner state so restored snapshots never skip new runs; verify scripts are staged only after the last step snapshot.
    • Backend: cloud_vm_env_layers table; workflows resolveEnvLayers/recordEnvLayer/listEnvLayers; routes /api/vm/env/layers and /api/vm/env/layers/resolve. Resolve derives depth from the caller’s chain and skips rows with mismatched stepIndex; upserts refresh stepIndex/baseImageId. Routes default provider to freestyle, validate input (sha‑256 hex chainHash/specDigest, stepIndex ≤ 255, capped lengths), and enforce team scope and snapshot ownership; typed errors env_provider_unsupported and env_layer_snapshot_not_owned. freestyle‑only support.
    • CLI ops: init, build, up, layers, logs; parser validates env keys as shell identifiers; client rejects empty snapshot_ids; build JSON reports finalLayerRegistered and the success banner only prints when the final layer is registered.
    • Internals: kept CLI/cmux.swift and Sources/Cloud/VMClient.swift at merge‑base length to satisfy the file‑length hard cap.

  • Migration

    • Add .cmux/env.yaml to your repo.
    • Use shell‑safe env keys ([A‑Z,a‑z,][A‑Z,a‑z,0‑9,]); no hyphens or digit‑leading.
    • Keep verify commands read‑only; they run before the final snapshot and their filesystem effects are cached.
    • Run cmux vm env build until verify passes; the final layer is only snapshotted and registered on success.
    • Use cmux vm env up for fast boots; if you change only verify, rebuild once to refresh the digest.
    • Use the freestyle provider for env builds.
    • Set base to a manifest image id; snapshot ids are not accepted as bases.

Summary by CodeRabbit

  • New Features
    • Added cmux vm env commands: init, build, up, layers, and logs, with per-step cached layers and --json reporting.
    • Introduced .cmux/env.yaml spec parsing and deterministic hashing for stable cache keys.
    • Added Cloud-backed env-layer endpoints and persistent snapshot caching to speed rebuilds.
  • Bug Fixes
    • Improved validation, error messaging, and ownership checks for env-layer resolution and registration.
  • Documentation
    • Added end-to-end docs for build/restore, caching behavior, and iteration loops.
  • Tests
    • Added spec/hash vector tests and Cloud env-layer integration tests (plus auth coverage).