acpmux: vault secrets reach Cloud agents as placeholders, filled in by a local egress proxy for allowed hosts only

Merged
TE
2d ago
+2036-68

Leo, 2026-11-03: "yeah let's do alllll of those things! :DDD" (one of them: agents on a Cloud machine should never see raw vault secrets).

What this does

Until now a Cloud machine's vault values went into the environment of every agent process there, so any agent, including one steered by a prompt injection, could read a secret and send it anywhere. With this change:

  • A vault entry delivered as proxy (the hub's new default, manaflow-ai/cmux-platform#742) reaches agents only as the placeholder NAME=cmux-secret://NAME.
  • acpmux runs a local egress proxy on 127.0.0.1 (new crate cmux-vault-egress) and points agents at it with HTTPS_PROXY. When a request header carries cmux-secret://NAME (Basic credentials included, as git sends them), the proxy puts the real value in, but only on a connection to one of NAME's allowed hosts.
  • Any other host gets a blind CONNECT tunnel, so the placeholder goes out unchanged and the secret never leaves the machine toward it.
  • Each substitution is logged as vault.egress_substituted with the secret's name and the host. Values are never logged.
  • An entry set to env delivery (cmux machine vault set NAME --env, from #34390) keeps the old behaviour of a real env var. acpmux host vault status labels it env (weaker: agents can read the value).
  • The machine asks the hub for GET /v1/vault/machine?delivery=proxy.v1. A hub answer of withheld (what an older machine gets for a proxy entry) is refused, never delivered.

Policy changes reach the next request: the proxy reads the vault's delivered set every 2 s at most, and a host-list or delivery change at the same revision updates the machine's record.

Two design questions, settled

TLS: a CA only agents trust, behind HTTP CONNECT.

  • The proxy serves only CONNECT; plain HTTP gets 405.
  • For a CONNECT to a host that some secret allows, it terminates TLS with a leaf for that host, signed by a CA made at daemon start. The CA key exists only in the daemon's memory.
  • Only agent processes are told to trust the CA, through their env: NODE_EXTRA_CA_CERTS, plus SSL_CERT_FILE, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE and GIT_SSL_CAINFO, each pointing at the system bundle plus the CA. Nothing is added to the system trust store, so no other process on the machine trusts it.
  • Upstream is a normal TLS connection verified against the OS roots, HTTP/1.1.
  • Every other host is never intercepted.
  • I rejected header injection on a plain HTTP proxy with no interception. An HTTPS request's headers are only visible inside TLS, so that design would need every agent to send plaintext to the proxy, which harnesses don't do.

Guards inside the intercepted path:

  • Domain fronting: a request whose Host or authority differs from the CONNECT target gets 421, so a shared front end can't route a value to another host.
  • DNS rebinding: destinations are resolved by the proxy, and loopback, private-link, unspecified and multicast addresses are refused, so a rebound name can't point a value at a local service.
  • Only header values are substituted. The URL and the body never are.

How harnesses pick it up.

  • Claude Code (Node) reads HTTPS_PROXY and NODE_EXTRA_CA_CERTS.
  • Codex (Rust, reqwest) reads HTTPS_PROXY.
  • Both harnesses' own model traffic goes to hosts no secret names (OPENAI_API_KEY and ANTHROPIC_API_KEY are reserved and cannot be stored in the vault), so that traffic is blind-tunneled and never sees the CA.
  • git, curl, Python requests, gh and npm read the variables above.
  • NO_PROXY keeps localhost,127.0.0.1,::1,.cmux.internal (Freestyle's alias domains) direct, merged with any inherited value.
  • If the machine already sets HTTPS_PROXY, acpmux keeps it, gives placeholders only and logs vault.egress_proxy_conflict once. Chaining to an upstream proxy is a follow-up.
  • The proxy starts only when some entry is proxy-delivered. If it fails to start, agents get placeholders with no proxy, so the failure is closed and the secret is not exposed.

Residual risk and follow-ups

  • An agent running outside the sandbox as the daemon's uid can still read the machine credential file and pull the vault itself. The bwrap sandbox already hides credential paths; separating that uid is a follow-up.
  • Inside the linux sandbox, its own proxy overrides HTTPS_PROXY, so placeholders stay unfilled there, which fails closed. Chaining the sandbox proxy to this one is a follow-up.
  • Not covered yet: websocket upgrades inside an intercepted connection, and substitution in the query string or body.
  • Go programs on macOS ignore SSL_CERT_FILE. Cloud machines are Linux, where they read it.

Tests

  • cargo test -p cmux-vault-egress runs a real TLS upstream behind the proxy:

    • an allowed host gets the value;
    • a denied host gets the placeholder through a blind tunnel, and the proxy CA is not even offered for it;
    • a second secret that allows another host stays a placeholder;
    • Basic credentials are substituted for git;
    • a Host mismatch gets 421;
    • a revoked secret stops on the next request;
    • plain HTTP gets 405 and an unknown host gets 502;
    • captured logs contain the name and host but never the value or its Basic encoding.

  • cargo test -p acpmux vault covers:

    • delivery and allowedHosts parsing, with withheld refused;
    • a same-revision policy change reaching the next spawn;
    • placeholders versus env values in the child env, and no proxy variables without a proxy entry;
    • a failed proxy start;
    • an inherited proxy;
    • NO_PROXY merging;
    • the trust files;
    • the state file holding hosts but no value.

  • The development E2E (machine-vault-e2e.yml, scripts/cmux-next/machine-vault-e2e.ts) now proves the proxy path. It runs vault set MACHINE_VAULT_E2E --host postman-echo.com, requires the new machine.vault.egress.v1 capability and proxy delivery in vault status, and has a Claude Code agent check three things: its env holds the placeholder, the allowed echo host got the nonce, and httpbin.org got the placeholder. Since #742 made proxy delivery the default, the old harness (which expected the raw value in the env) could no longer pass.

Observed / Not run

  • Fleet run at b4e7ae5 (Linux pool):
    • cargo test -p cmux-vault-egress: 8 unit, 5 egress and 1 log test pass.
    • cargo test -p acpmux -- vault capabilit: 63 + 1 pass.
    • cargo clippy -p acpmux -p cmux-vault-egress --all-targets: no warnings in changed files. The one warning is in hub/permissions/repo_sync_tests.rs, which is on base.
    • cargo fmt --check: clean.
    • bun test scripts/cmux-next/machine-vault-e2e.test.ts: 26 pass.
  • Not run yet: the development E2E (machine-vault-e2e.yml) on a real joined machine. It installs acpmux from the published feat-cmux-next channel, so it runs after the land, and its result goes in a comment here.

Changelog

Added: Cloud machine vault secrets reach agents as cmux-secret://NAME placeholders, filled in by a local proxy only for each secret's allowed hosts.

Agent session: agent session

šŸ¤– Generated with Claude Code