PR #27603 - manaflow-ai/cmux

Leo (2026-10-29): "Use cmux Cloud VMs: start a lane to ship cmux-vm in the NEXT bundle and let agents (including ones outside cmux, with a scoped token) run cmux vm run --sync."
Leo (2026-10-19): "anybody logged in with an @cmux.com account ALREADY CAN JUST USE THE VMS"
Leo (2026-10-29), on landing #27604 as designed (Labs-gated, VM-only scopes, up to 24h, budget-capped): "Yes, ship it"

What was wrong

cmux vm … in the NEXT bundle exec'd a sibling cmux-vm binary that no bundle ships, so every verb (even --help) failed with cmux-vm not found in PATH. Agents outside cmux had no supported way to get a VM at all, so they fell back to an unshared Freestyle key.

What changes

cmux vm runs in process. cmux-tui links the cmux-vm CLI crate (workers/cmux-vm/cli) by path and runs it on its own tokio runtime. The crates gain a default-tls feature; cmux-tui turns it off and rides its existing rustls + ring, so no aws-lc enters the bundle (cargo tree -i aws-lc-sys is empty). tree workers/cmux-vm/cli joins the cmux-tui tree key.
Packaging check. cmux vm hands cmux-vm the words after vm verbatim (the global scanner would read --sync as taking a value and eat the --).
scripts/cmux-next/check-bundled-cli-commands.sh runs every line of scripts/cmux-next/bundled-cli-commands.txt with the bundled bin/cmux, an empty environment and PATH=/usr/bin:/bin, and fails on a nonzero exit or a "not found in PATH" / "cannot start" message. The nightly-next signed-app verify job (macOS) runs it on the variant whose slice that Mac can execute, so a subcommand that hands off to a binary the bundle lacks fails before publish. A unit test also runs vm --help, vm run --help and vm token --help in process with no cmux-vm anywhere.
cmux vm run [--sync] [--dir D] [--pull PATH]… [--keep] [--idle-timeout S] [--timeout S] -- CMD.
- Creates an ephemeral VM (label cmux.run=1, idleTimeoutSeconds default 900, autoDeleteSeconds 900 unless --keep) and waits for it to run.
- --sync uploads the tree to /work: git ls-files --cached --others --exclude-standard (a directory walk outside git), never .git, node_modules, target, build output, anything under a credential directory (.ssh, .aws, .gnupg, .docker, .kube, .azure, .gcloud, .terraform.d, .password-store, .vercel, .config/{gcloud,gh,op,doctl,hub}), anything under a symlinked directory (tar would follow it out of the tree), or credential-shaped files (.env* except examples, *.pem, *.key, id_*, .npmrc, .netrc, .git-credentials, credentials.json, the cmux VM config and token files). 64 MiB cap. The tarball goes over exec stdin in 1 MB chunks, since agent keys have no vm:files scope.
- Streams stdout and stderr, returns the command's exit code (124 on --timeout, 130 on Ctrl-C/SIGTERM).
- --pull PATH copies paths back, refusing archive entries that escape the target (absolute, ..), archives over 256 MiB and anything that unpacks past 2 GiB (counted locally while it streams; the guest's numbers never size an allocation).
- Owns the VM from the moment create answers: a boot timeout, a polling error, Ctrl-C or SIGTERM (handlers are in place before create) all end in delete, or pause with --keep; the last stderr line names the VM and deleted or paused. 429/503 are retried with backoff.
cmux vm token create|status|revoke.
- create gets a vm.cmux.dev agent key. With no credential of its own (no CMUX_VM_API_KEY, no config file) it asks the cmux app over the socket (vm.agent_key.create, #27604); the app mints with the signed-in session. The key goes straight into the login Keychain (com.cmux.vm.agent-token; a 0600 file under $XDG_CONFIG_HOME/cmux off macOS, refused if group/other-readable) and is never printed, in text or --json.
- From then on any process of this user, inside or outside cmux, can run cmux vm …: precedence is CMUX_VM_API_KEY > --config/CMUX_VM_CONFIG > stored token > ~/.config/cmux/vm.json. A stored token that is expired or cannot be read is refused, never silently replaced by the default config file's key (which may be a broader team key).
- status shows id, scopes, budget and expiry (exit 11 when expired). revoke deletes the key server side (which deletes its VMs) and the local copy.

Companion PR (the app side, Labs-gated): #27604.

Design and threat model

Asset: the ability to create and run code on cmux Cloud VMs billed to the user's team.

What the token is. A vm.cmux.dev agent key, a server feature that already exists (POST /v1/agent-keys). The server, not this client, enforces:

scopes vm:read, vm:write, vm:exec only: no files, network, terminal, snapshot or admin routes;
access only to VMs the key itself created (label cmux.agent=<id>), never the team's other VMs;
a budget: maxVms (CLI default 3, cap 20) and maxUsdPerHour (default $2, cap $50);
a TTL (CLI default 8h, max 24h). Revocation or expiry deletes the key's VMs;
an audit row on creation (agentkey.create). An agent key cannot mint keys.
The CLI always sends exactly the three VM scopes; --scope is refused.

Who can mint. Only a process the cmux socket already trusts (its same-process rule is unchanged), through an app that is signed in, has a team, and has Labs > Show Preview Features on. The Stack session never leaves the app; the CLI only ever sees the agent key.

Who can use it. Any process running as the same macOS user that can read the login Keychain item (the item's ACL is the cmux binary that wrote it; another binary prompts the user) or holds CMUX_VM_API_KEY. This is the point of the feature: agents outside cmux. It widens nothing else: every other socket method keeps the same-process rule, and the token is useless outside the VM routes.

Threats and answers.

Token leaks (logs, transcripts, --json output): it is never printed; Debug is redacted; status shows only the key id. Worst case is bounded by scope, budget, own-VMs confinement and TTL, and cmux vm token revoke ends it and its VMs.
Credentials reaching a VM: sync skips credential-shaped files, credential directories and anything under a symlinked directory; the token itself is never sent to the guest (exec env carries only the command).
Hostile guest on pull: download and unpacked sizes are bounded locally; entries are listed and checked before extraction (no absolute paths, no ..), --no-same-owner.
Silent authority change: an unusable stored token fails closed instead of falling back to a broader key.
Runaway cost: per-key VM and $/hour budgets, idle timeout, auto delete, delete on exit and signals.
Lateral access to other team VMs: refused by the server's own-VMs rule.

Not covered here: the key is a bearer token; a same-user process with Keychain access can use it. That is the same trust as ~/.config/cmux/vm.json API keys today.

Size

About 1,600 hand-written lines plus the lockfile. run and token are one feature with their tests; the app method is split into its own PR.

Testing

cargo test -p cmux-vm: 12 unit + 80 + 1 integration tests pass, clippy clean. New tests cover token precedence, expiry fallback, TTL parsing, the create body, redacted Debug, run's command quoting, credential-shaped skips and pull-path checks.
cmux-tui cli::vm tests: plan mapping for run/token, the app-mint decision, mint params, and --help in process with no cmux-vm binary. Run through cmux-ci on this PR.
End-to-end proof (a NEXT nightly running cmux vm run --sync -- bun --version from a plain Terminal process with the stored token) follows once both PRs are in a nightly.

CI wiring: workers/cmux-vm/cli/** joins the cmux-next paths filter (the router now reads it); the notices treat its crates as first-party and carry progenitor-client's (MPL-2.0) reviewed license text.

Review follow-ups

A create whose answer is lost (network error, 5xx, or a 2xx naming no VM) is read back by its idempotency key: committed, the run goes on with that VM and ends it as usual; pending, it is cancelled; anything not confirmed keeps the key and names the creation get / creation cancel / delete commands. cmux vm creation is reachable from the bundled CLI, and the nightly bundle check (both signing paths) runs it.
exec retries only on 429, never on 503, so a command or an upload chunk never runs twice.
--ttl saturates instead of wrapping.
Errors and hints name cmux vm when cmux runs cmux-vm in process, cmux-vm otherwise; a 404 says what was not found and the verb that lists them.
The no-API-key hint names what cmux vm token create needs: a cmux terminal with the app signed in and Labs on. It does not name cmux cloud sign-in, since cmux has no CLI sign-in verb; sign-in happens in the app.

Overlap: #25367 (Cloud provider parity) touches the same workers/cmux-vm/cli files and cli/vm.rs; whichever lands second resolves.

Review follow-ups after f199ac12 (Cloud area, which adopted this PR)

Fixes for the review at f199ac12 (BLOCK, 3 findings) and the reviews after:

cmux vm token create through the app left the minted key live when its answer could not be read or stored. It now goes through cmux_vm::token::keep_minted, which stores the key or revokes it with its own key (an agent key may revoke itself), as the API path's revoke_unkept already did.
token status exited 0 for a token with no readable expiry. StoredToken::status now refuses it like an expired token (exit 11), as resolution does.
Each guest poll waited a fixed 20 s before the --timeout check (--timeout 1 could run about 21 s). The wait is now capped by the time left (poll_wait_tenths).
token revoke --json printed text after the revoke had happened. It now prints {"id","revoked","removed"}.
A run create that ended in a 429 for capacity exited with no recovery and no key, although the server keeps a VM whose growth failed after allocation, uncommitted, under the create's key (src/handlers/vms.ts). It now goes through recover_creation. 429s for rate, vms and the agent key's budgets refuse before anything is allocated, so they still end the run as before.
token create overwrote a usable stored token, leaving its key live and billable with nothing able to revoke it. It now refuses (exit 15) while a usable token is stored and names token revoke (which deletes its VMs) as the way to replace it. It never revokes on its own, since another process may be using those VMs. An expired token or an unreadable store is still replaced. Both mint paths check before minting.
App mint errors now keep the VM CLI's exit codes: not signed in 11, plan required 12, forbidden 13, rate limited 16, no answer 3, 5xx 18.
token revoke revoked at the stored token's own issuer only after a flag or CMUX_VM_BASE_URL; a stored key now goes only to the service that minted it, for every command, and another base URL is refused.
token create holds a per-user store lock (also taken by token revoke), so concurrent creates and revokes cannot orphan a key or drop a newer one.
run: the team's idle timeout by default (--idle-timeout -1 never pauses), and the service's own maxRunSeconds (--timeout + 30 min) bounds every run so a dead client leaves no VM running; a failed --timeout kill is reported; reading a finished command's output is never a timeout; cleanup failure keeps the command's exit status.
--sync: hard-linked files, files under symlinked directories and credential-shaped names (UTF-8 or not) never sync; non-UTF-8 git paths sync; a worktree git cannot list is refused, never walked; the 64 MiB cap is enforced while tar streams. --pull stops on the guest one byte past its 256 MiB cap and removes its archive.
Expiries are parsed exactly (fixed-width RFC 3339, real dates); a token with no readable expiry is refused like an expired one.
Linked in process, the cmux CLI no longer takes cmux-vm's reqwest system-proxy (SystemConfiguration, missing from the Linux cross link's framework stubs) or serde_json's preserve_order (which Cargo would have unified into every serde_json map of the CLI, and which made whole-workspace clippy fail with result_large_err in unrelated crates). Both are now default features of the standalone cmux-vm binary only; in the cmux CLI HTTP(S)_PROXY still applies and cmux vm --json prints the same fields with sorted keys. scripts/ci/macos-stubs/Security.tbd and CoreFoundation.tbd export the keychain symbols security-framework's generic-password calls import. The cmux-tui artifacts run at 5fcdf6a0 builds both macOS targets on Linux.
--sync matches credential directories in any case (.AWS/config, .CONFIG/GH/hosts.yml on a case-insensitive volume).
The --timeout kill lists the command group's live members from /proc and signals each (dash's kill takes no negative pid), never signals pid 0 or 1, and reports a group it cannot find or end.
A VM answer without a state ends the run at once (the VM is deleted) instead of polling a billable VM for the boot deadline.
--pull refuses a path whose parent resolves outside the work directory through a link the command made (leak -> /etc, then --pull leak/passwd); a path that is itself a link is archived as the link.
Linked into the cmux CLI (minimum Rust 1.91), the cmux VM CLI crates declare 1.91 too; both workspaces pass cargo check --locked on 1.91.0. generate-third-party-notices.sh in pin mode archives and scans workers/cmux-vm/cli only when the cmux-tui pin has it (the current pin predates it).
--sync leaves out the coding agents' auth stores (Claude's .credentials.json and .claude.json, Codex's .codex/auth.json, opencode's opencode/auth.json, Gemini's oauth_creds.json), judges credentials by the synced directory's own path too (--dir ~/.codex, --dir ~/.config/gh send none), and skips node_modules, target, .git and the rest in any case.
cmux vm token create with --base-url or --team (either form, --flag value or --flag=value) runs in the cmux VM CLI in process, like --config and --help, instead of going to the app's mint, which mints only at the app's own signed-in issuer and team.
Merged with #28280's --idempotency-key forwarding: since the words after vm reach cmux-vm verbatim here, a key typed after vm (before --, either form) is read like --json and handed to the verbs that take one (create, fork, snapshot create, and now creation get|reconcile|cancel, which require it and lost it before); it is dropped for other verbs (creation list included), as on base, and #28280's tests pass unchanged. A key flag with no value never takes the -- after it.

Follow-ups, to be filed as issues: --sync caps the compressed archive but not the extracted size, and sync and pull archives can stay on a VM kept with --keep after a failed extraction or cleanup (review at 822c992b, https://github.com/manaflow-ai/cmux/pull/27603#issuecomment-6546811441); the macOS token lock is keyed on the config directory while the Keychain item is per user (review at 69477032); an unreadable stored token's error names token revoke, which fails the same way (review at f418f98e). cmux vm run takes a command longer than the API's 32 KiB env value limit and many --pull paths can exceed its 64 KiB command limit, each failing only after the VM is made (review at 2de5647e, https://github.com/manaflow-ai/cmux/pull/27603#issuecomment-6548078887). A cmux-vm global option between nested verbs (creation --team T get) drops the forwarded key (review at f1946e17, https://github.com/manaflow-ai/cmux/pull/27603#issuecomment-6548254645). Found by these reviews in code this PR does not change (workers/cmux-vm/src, from the agent-keys PR), also to be filed: agent-key budgets are not checked on resize, and the budget check is not atomic with create.

Known limit: the service's POST /v1/agent-keys takes no idempotency key, so a mint whose answer is lost cannot be found again by the client. Both mint paths say so; such a key was never shown, has no VMs and expires within its --ttl. Closing it needs a service change.

New tests: run::tests::credential_directories_match_in_any_case, the_kill_script_reports_a_command_it_cannot_find_or_kill, a_pull_never_leaves_the_work_directory_through_a_link, lifecycle::run_fails_fast_and_deletes_a_vm_whose_answer_has_no_state, agent_credentials_never_sync, syncing_a_credential_directory_itself_sends_none_of_it, skipped_directories_match_in_any_case, the --base-url/--team cases of cli::vm::tests::token_create_with_an_explicit_issuer_or_help_runs_in_process, cli::vm::tests::creation_list_takes_no_key, a_key_without_a_value_never_takes_the_double_dash (each red at its test-only commit 0a5f470f, 6b216fc5, fcf44848, 364785fc, 6029492e or eb3badae on cmux-ci, green at the head); token::tests::an_app_minted_key_that_cannot_be_kept_revokes_itself, status_refuses_an_unreadable_expiry_like_an_expired_one, revoke_json_is_one_json_object; run::tests::a_poll_never_waits_past_the_deadline and the extended unconfirmed cases; token::tests::create_refuses_while_a_usable_token_is_stored; cli::vm::tests::app_mint_errors_keep_the_vm_cli_exit_codes.

Reds at f199ac12 on #27603 were base: the sidebar-look sweep named #24073, #24295 and #27561 (all on base), and the literal-state-roots red is in acpmux/src/cli/help_examples/session.rs, which this PR does not touch.

No capture: CLI only, no UI change.

Observed / Not run

Observed: cargo fmt --check clean for workers/cmux-vm/cli; a rustc probe of components().map(Component::as_os_str) (one review claimed it does not compile) compiles and runs.
Observed at e14ca7d0 (cmux-ci run 41150195430): cargo test -p cmux-vm 38 unit + 81 integration + 1 pass; cargo clippy -p cmux-vm --all-targets -- -D warnings clean; cargo test -p cmux-tui cli::vm 20 pass.
Observed at 80696572 (cmux-ci run 41235558986): the panic ratchet on the changed cmux-tui packages (cmux-tui-clippy-changed.py --panic-ratchet against base 8769d23b) passes; at f1946e17 it counted two new indexing_slicing hits in split_raw (833 > 831), which 80696572 removes. cargo test -p cmux-tui cli::vm 25 pass; whole-workspace clippy in cmux-tui clean.
Observed at f1946e17 (cmux-ci run 41234898271), after merging feat-cmux-next 8769d23b: creation_list_takes_no_key and a_key_without_a_value_never_takes_the_double_dash fail at the test-only commit eb3badae and pass at the head; cargo test --locked -p cmux-vm 44 unit + 82 integration + 3 (base's agent guide) + 1 pass; cargo test -p cmux-tui cli::vm 25 pass (base's three --idempotency-key tests unchanged); clippy for cmux-vm with and without default features and whole-workspace clippy in cmux-tui clean.
Observed at 2de5647e (cmux-ci run 41232229186): cargo test --locked -p cmux-tui token_create_with_an_explicit_issuer fails at the test-only commit 6029492e (the --base-url case goes to the app's mint) and passes at the head; cargo test --locked -p cmux-vm 44 unit + 82 integration + 1 pass; cargo test -p cmux-tui cli::vm 20 pass; whole-workspace cargo clippy --locked --keep-going --workspace --all-targets -- -D warnings in cmux-tui clean.
Observed at 8985a08b (cmux-ci run 41230352385): cargo test --locked -p cmux-vm 44 unit + 82 integration + 1 pass; cargo clippy -p cmux-vm --all-targets -- -D warnings clean, also without default features; cargo test -p cmux-tui cli::vm 20 pass; whole-workspace cargo clippy --locked --keep-going --workspace --all-targets -- -D warnings in cmux-tui clean (it failed at e14ca7d0 and passes at the merge base).
Observed at f418f98e (cmux-ci run 41229997390): cargo check --locked on Rust 1.91.0 passes for cmux-tui (whole workspace) and the cmux VM CLI workspace (all targets). THIRD_PARTY_LICENSES.md regenerated in pin mode is stale at the head and at the base tip c51a4a91 alike (every stale line at the head is stale on base too), so that red is base's.
Observed at 5fcdf6a0: the cmux-tui artifacts run (41221086004) builds aarch64- and x86_64-apple-darwin on Linux; nothing after it changes what is linked.
Not run: an end-to-end cmux vm run --sync against the live service from a NEXT nightly.

🤖 Generated with Claude Code
Agent session: agent session