PR #11789 - manaflow-ai/cmux

What

The cmux app can now own the WireGuard tunnel into the user's private Cloud VM network as a real macOS VPN, through a bundled NetworkExtension packet-tunnel system extension. On a build whose signing carries the capability, opening a Cloud machine brings the tunnel up with no cmux vpn up, no sudo, no wg-quick, no Homebrew; the tunnel stays off until the user actually uses cmux Cloud and stops again when no Cloud sessions remain. Builds without the capability fail closed for Cloud browser access; terminal traffic continues through the user-space WireGuard hub.

Closes #11760

Architecture (and the two places this deliberately departs from the issue text)

System extension, not app extension. The issue asked for a packet-tunnel-provider app extension. macOS only loads NetworkExtension app extensions for apps distributed through the Mac App Store; cmux ships as a Developer ID app (DMG, Homebrew cask, Sparkle). An appex would build and sign but never load. The provider is therefore a network system extension (cmuxTunnel.systemextension, bundle id <app bundle id>.tunnel, embedded at Contents/Library/SystemExtensions) activated with OSSystemExtensionRequest, and the entitlement value is packet-tunnel-provider-systemextension plus com.apple.developer.system-extension.install. Two consequences the user will see once the capability ships: the first activation on a Mac asks for approval in System Settings › General › Login Items & Extensions, and macOS refuses to load system extensions from an app outside /Applications (the app reports that as an actionable error).

Config transport is NETunnelProviderProtocol.providerConfiguration, not an App Group container. System extensions run as root and cannot read the user's group container or keychain. The completed wg-quick config (private key filled in) travels inside the VPN configuration the app saves, which lives in the root-only NetworkExtension preference store, strictly more protected than the 0600 file the wg-quick path keeps. The team-prefixed App Group (7WLXT3NR37.com.cmuxterm.app[.nightly]) is still declared on both bundles because the extension's NEMachServiceName must be prefixed by one; team-prefixed groups need no portal registration.

Extension target (`TunnelExtension/`, target `cmuxTunnelExtension`)

PacketTunnelProvider (NEPacketTunnelProvider) reads the wg-quick config from providerConfiguration, parses it, and runs it through WireGuardKit's WireGuardAdapter. Config parse errors are logged by rule name only (the private key is one of the payloads). handleAppMessage answers the runtime configuration (peers, handshakes, transfer) with the private and pre-shared keys stripped.
WireGuardKit is vendored at vendor/WireGuardKit (MIT; upstream wireguard-apple 2fec12a6, tag 1.0.16-27) as a local SwiftPM package, with the upstream app target's wg-quick parser moved into the kit and made public, and one header fix for Xcode 26's strict module imports. vendor/WireGuardKit/README.md records every modification and the update procedure.
wireguard-go is compiled into the extension by a new build phase, scripts/build-wireguard-go.sh (per-arch go build -buildmode=c-archive in readonly module mode, lipo'd). Release builds require Go and fail without it (the release workflows install it); Debug builds without Go, including the Debug CI lanes and fleet dev builds, get a loud stub archive (every wg* symbol fails, marker symbol cmux_wireguard_go_bridge_is_stub), and release signing refuses to ship a stub. Upstream's Go-runtime patch (timers counting sleep, an iOS concern) is intentionally not applied; the adapter re-handshakes via its network-path monitor after wake.

App side (`Sources/Cloud/Tunnel/`)

CloudTunnelBackendSelector picks the backend from the running binary: signed packet-tunnel-provider-systemextension + system-extension.install + a .systemextension actually present in the bundle → NetworkExtension; anything missing → unavailable with a named reason. VMTunnelManager.networkExtensionAvailable() now delegates to it.
CloudTunnelCoordinator (actor) is the state machine: off → starting → (awaitingApproval) → up → stopping → off, failed(message). It enrolls (idempotently) through VMTunnelManager, saves the VPN configuration through NetworkExtensionTunnelController (NETunnelProviderManager, isOnDemandEnabled = false, no on-demand rules), activates the extension through SystemExtensionActivator, and waits for NEVPNStatus to reach connected. Because the system extension outlives the app, a tunnel left connected by a previous instance is adopted on the next Cloud use instead of restarted, and quit / sign-out / cmux vpn down still stop it before any Cloud use (the controller reads the existing VPN configuration at launch, a passive preferences load with no prompt). A start superseded by a stop (an approval wait is not cancellable) can never tear down a newer start's tunnel; a Cloud use that arrives while a stop is draining queues behind it instead of racing NetworkExtension; and after a failed start Cloud uses back off for 30 s instead of re-running enrollment, activation, and the configuration save on every dial (cmux vpn up always retries).
On-demand trigger: VMClient takes a CloudPrivateNetworkGate; every endpoint-minting call (openSSH, openAttach, openCmuxRemote, openSession, openPort) starts the tunnel concurrently with the HTTP request and waits a bounded readiness budget (20 s) before returning the endpoint, so the dial finds the route in place. This is the single funnel for the Machines panel, headless cmux-tui links, session restore, and every CLI vm verb. Nothing runs at launch.
Idle stop: while up, an idle timer (5 min) restarts on every Cloud use; when it fires with no live consumers (workspaces bound to a Cloud machine across all windows + connected cmux-tui links) the tunnel stops. Sign-out, vm.tunnel_revoke, and app termination stop it too. cmux vpn up pins it until cmux vpn down.
Composition root is AppDelegate (makeCloudTunnelCoordinator), injected into VMClient.bootstrap and TerminalController for the socket verbs.

CLI and socket

vm.tunnel_status/vm.tunnel_config now report backend, tunnel_state, pinned, extension_bundle_id, and addresses; interface_up comes from NEVPNStatus on the Network Extension backend. New verbs vm.tunnel_up, vm.tunnel_down, vm.tunnel_wait (long-poll on the state stream, no sleeping); The Machines sidebar's private-route blocker text reads the coordinator's state on the app-managed backend (starting / waiting for approval / failed / down) instead of telling the user to run cmux vpn up.
cmux vpn up|down|status|revoke pick their path from backend: app-managed builds never touch sudo/wg-quick (up reads the status, lets the app's start enroll once, pins, and waits through the first-run approval with vm.tunnel_wait; down releases); unavailable builds fail closed for browser access. Help text updated; new strings localized (en/ja).

Release pipeline

cmux.release.entitlements / cmux.nightly.entitlements declare the desired state: the NetworkExtension capability, system-extension.install, and the App Group.
scripts/reconcile-entitlements-with-profile.py (+ tests) computes the effective entitlements against the embedded provisioning profile; scripts/sign-cmux-bundle.sh reconciles the app entitlements against the embedded profile and fails the release if the requested tunnel capability is not granted. When the profile grants it, the script requires the extension's own embedded profile, rejects a stub bridge (scripts/verify-tunnel-extension-engine.sh checks for the Go __go_buildinfo section and an exported wgTurnOn, both of which survive stripping), signs the extension with TunnelExtension/cmuxTunnelExtension.{release,nightly}.entitlements, and verifies both sides agree.
Workflows: actions/setup-go before the Release xcodebuilds (release, nightly, CI release-build); required APPLE_{RELEASE,NIGHTLY}_TUNNEL_PROVISIONING_PROFILE_BASE64 secrets embedded by scripts/ci/embed-tunnel-extension-profile.sh (+ tests); nightly's prepare_variant renames the extension to com.cmuxterm.app.nightly.tunnel; architecture checks cover the extension binary; strip-release-bundle.sh strips it.
Third-party notices for WireGuardKit, wireguard-go, and golang.org/x added to THIRD_PARTY_LICENSES.md.

Apple Developer and CI signing state

Team 7WLXT3NR37. This setup is complete.

Active Developer ID profiles exist for com.cmuxterm.app, com.cmuxterm.app.nightly, com.cmuxterm.app.tunnel, and com.cmuxterm.app.nightly.tunnel, with Network Extensions/System Extension capabilities enabled.
GitHub Actions contains the four matching profile secrets used by the release and nightly workflows.
The release and nightly workflows install Go, embed the extension profile, verify the real WireGuard engine, sign the nested extension and app, smoke-test, and notarize.

The profiles are managed in Apple Developer › Profiles and the encrypted values are managed in GitHub Actions secrets.

Trade-offs stated

System extension over app extension (above): correct for Developer ID, but adds the one-time System Settings approval and the /Applications requirement.
Private key in providerConfiguration: root-only store, better than today's 0600 file, but the key is now held by the system rather than only by files the app owns.
Go toolchain at build time: Release builds and CI need Go (actions/setup-go added); Debug builds without Go silently get a non-functional stub extension that cannot load anyway (no entitlement) and cannot ship (signing rejects it).
Idle stop counts app-visible sessions only (Cloud workspaces + links) after a 5-minute grace. A raw ssh from another terminal app using cmux vm ssh-info is not visible to the app and could lose its route after that grace; cmux vpn up pins the tunnel for that case.
.internal hostnames still need cmux vpn hosts (sudo) on both backends; DNS through the tunnel is a follow-up.
New strings are en + ja, matching the existing cli.vpn.* coverage.
Tagged Debug builds embed the extension under the app's own bundle id. reload.sh overrides PRODUCT_BUNDLE_IDENTIFIER for every target, so a dev build's extension reports the app id (as the Dock tile plugin already does). Harmless: Debug builds carry no entitlement and never activate the extension. Release and nightly builds get <app id>.tunnel.
Fleet/CI Debug builds without Go ship a stub engine in the extension; the PR verified this path on the fleet (builders have no Go) and the real engine locally (Go 1.26).
Merging main (#11762/#11773 and later) required a compile fix outside this feature. #11773 gave MachineCreateCoordinator.Launch a progress parameter and updated NewMachineSheetPresenter but not the Base-open launcher in AppDelegate.swift, so main itself no longer compiles the app target (every app-host CI lane on main is red). This PR carries the two-line call-site fix as its own commit so its CI can be green; if main lands the same fix, the merge resolves trivially.
Structured review ran on the Claude engine. The default Codex engine was unavailable on this Mac (the local Codex proxy's account needs re-authentication, HTTP 503, twice); the canonical helper was rerun with --engine claude. Its accepted findings (adopting an already-connected tunnel; generation-guarding the superseded start's cleanup; stopping an inherited tunnel on quit/sign-out/down; a failed-start backoff; queuing a mid-stop Cloud use behind the stop; failing fast when an adopted connecting link drops; redacting keys from the runtime-configuration reply; a single enrollment per cmux vpn up; a strip-proof stub check; eager pruning of departed status subscribers) are fixed with regression tests; the one-type-per-file policy findings are fixed by splitting and by lifting nested types to their own files. Two findings are rejected: the DispatchQueue use in vendored Array+ConcurrentMap.swift (upstream third-party code kept byte-identical), and "store the WireGuard private key in the keychain via passwordReference" (a system extension runs as root and cannot read the user's keychain; the root-only NetworkExtension preference store is the correct and strictly stronger home for it, see the transport note above).

Verified

Current verification state:

Fleet Debug build (reload-cloud.sh --tag issue-11760-vpn-networkextension) succeeds: the extension target compiles and links, cmuxTunnel.systemextension is embedded at Contents/Library/SystemExtensions with CFBundlePackageType = SYSX, NEProviderClasses → cmuxTunnel.PacketTunnelProvider, and localized InfoPlist.xcstrings (en/ja). The fleet builders have no Go, so that build carries the stub engine as designed; the real engine was built locally with Go 1.26 as a universal libwg-go.a exporting wgTurnOn (about 30 s).
Live dogfood of the unentitled path on the tagged Debug app, first with the dev backend disabled and then rebuilt from the merged head with --prod-auth against the real Cloud service (this Mac has a wg-quick tunnel up): vm.tunnel_status reports backend: wg-quick, fallback_reason: entitlement-missing, tunnel_state: off, interface_up: true (address-based liveness still detects the running wg-quick interface); vm.tunnel_up answers with the localized "this build does not include the app-managed tunnel" error; vm.tunnel_wait/vm.tunnel_down are no-ops; cmux vpn status output is unchanged (Tunnel: up, Backend: wg-quick); system.capabilities lists the six vm.tunnel_* verbs; no VPN configuration appears in scutil --nc list. Nothing starts at launch.
Real signing runs with the Manaflow Developer ID certificate on copies of the built app (CMUX_TIMESTAMP=none): without an embedded profile the script drops the three tunnel entitlements, removes the extension, and the app verifies with no NetworkExtension keys; with synthetic profiles granting the capability it keeps the extension, signs it with the release entitlements (packet-tunnel-provider-systemextension, App Sandbox, App Group, network client/server) under the Developer ID, and the app's signature carries the capability. Both runs then stop at the pre-existing universal-sidecar check, which a Debug (arm64-only) app cannot pass; the tunnel-specific tail checks were repeated by hand.
Script tests pass locally: tests/test_reconcile_entitlements_with_profile.py (8), tests/test_embed_tunnel_extension_profile.sh, tests/test_strip_release_bundle.sh, plus the existing notarization/workflow guard tests. tests/test_ci_universal_release_settings.sh fails on main already (setup.sh no longer carries the string it greps) and is untouched.
Unit tests (CloudTunnelCoordinatorTests incl. adoption, inherited-tunnel stop, failure backoff, mid-stop queuing, fast-fail on drop, and superseded-start cases; CloudTunnelBroadcastTests; CloudTunnelRuntimeConfigurationRedactorTests; CloudTunnelBackendSelectorTests, existing VMTunnelManagerTests/VMTunnelStalenessTests) run in CI's app-host lane; the fleet's xctest lane needs a GUI console session and no builder passed its preflight tonight. The first CI run on the merged head failed for two reasons that are both fixed here: the Debug lanes have no Go (the script now requires Go for Release only) and main's AppDelegate compile error above.
Signed artifact validation: a non-publishing Nightly fast dogfood run is in progress on this branch at Actions run 33931128749. It is the first end-to-end check of the active profiles, extension embedding, Go engine verification, and signing path; it does not update the shared Nightly release or feed.
Runtime acceptance: after the signed artifact is installed under /Applications, opening a Cloud browser should trigger the one-time System Settings approval, then cmux vpn status should report the Network Extension backend and Tunnel: up; cmux vpn up/down remain optional manual controls.

🤖 Generated with Claude Code

Summary by CodeRabbit

New Features
- Added automatic, app-managed Cloud networking that starts when private machine access is needed and stops when no longer in use.
- Added tunnel installation, approval prompts, connection status, pinning, and lifecycle management.
- Added cmux vpn support for app-managed tunnels; builds without the signed extension fail closed for browser access.
Improvements
- Added clearer tunnel status, errors, progress messages, and Japanese localization.
- Added privacy messaging for network extension access.
Documentation
- Updated private networking and CLI help documentation.