Computer Use: per-app approvals on Mac, iPhone, Windows and Linux (cua-driver)

AZ
3d ago
+61331-707

Lands after #28584 (cmux Remote). This branch is stacked on it, so the diff below includes #28584's commits until that merges.

Summary

Agents can use apps on macOS, Windows and Linux through upstream Cua Driver (trycua/cua, MIT, pinned cua-driver-rs-v0.34.0 / b0968e1).

  • Default mode: Standard (D24). computerUse.mode defaults to standard, upstream's own default: no per-app question. "Ask Before Using Apps" is one setting away. Never-allowed apps (password managers, auth prompts, System Settings, terminals, cmux) stay refused in every mode.
  • Setup asks on enable (D25, D26). Turning Computer Use on, from any entrypoint, shows a user-only setup card:
    1. Device Control and Data Access (macOS 27's name for Accessibility) for cmux Remote.
    2. Screen & System Audio Recording for cmux Remote.
    3. On a Mac joined to a team: Screen sharing and Control (Settings > Devices). acpmux refuses every call with consent_required without them. The card says these switches also let the team's machines see and control the Mac. Turn On asks for Touch ID for each switch, through the switches' own person path. A later revoke in Settings brings no card back.
  • In Ask mode, the question. The first time an agent session wants an app, a cmux card asks: Allow Once, Allow for This Session, Always Allow , or Deny. The same question goes to the iPhone, and the first answer wins.
  • Timing. An unanswered call fails after 2 minutes; the question stays answerable for 10.
  • Enforcement. Upstream Cua Driver enforces every answer. Each agent session is an upstream bounded session whose app list grows only with the person's answers. No second policy engine.
  • Discovery before approval (D21). Agents can list running apps and windows with metadata only (names, ids, pids; never titles or pixels), so they can reach the question.
  • Windows and Linux. Same architecture as macOS: a Rust cmux Remote host (cmux-tui/crates/cmux-remote-host) on the same remote-host-v1 wire and driver commit. It has polkit and Windows Hello collectors, X11 and Wayland, and an interactive-session task on Windows.
  • VMs and own boxes. The owner sets the mode once with cmux computer-use setup (elevated): unrestricted for disposable cmux VMs (upstream's two-part opt-in), bounded for a user's own box.
  • Remote approvals.
    • The device hub carries approvals to the iPhone. A box accepts an Allow only from an iPhone approval key whose fingerprint the owner pinned. That key is in the Secure Enclave and needs Face ID for every signature.
    • The iPhone shows its fingerprint in Settings ā†’ Computer Use. The hub's word alone never grants.
  • Controls.
    • MCP: cua_* tools plus cua_status and cua_approvals_list|revoke.
    • CLI: cmux computer-use status|tools|call|screenshot|approvals|setup.
    • Palette: Computer Use: Show Status, and Revoke Saved App Approvals.
    • Settings: computerUse.mode and computerUse.phoneAlwaysAllow (default off).
    • CLI and MCP can only lower trust; raising it needs a person.

Design docs: docs/computer-use-approvals.md, docs/cmux-remote.md (Windows and Linux). Trust notes with recommendations: https://github.com/manaflow-ai/cmux/issues/13742#issuecomment-6623818743. Step 3's trust note: https://github.com/manaflow-ai/cmux/issues/13742#issuecomment-6648727003. The 26 design decisions (interview plus later ones) are in the proof report.

Testing

Head 3cb22674.

CheckWhereHeadResult
Fleet app build, tag cmux-next-computer-usejob 054da13330567160b5b5ae733cb22674passed, published to HQ
Setup step 3 (team consent), redstep 0b039252023b0dff4966d96d962ea3e6red: the team step, card and Turn On button don't exist
Setup step 3, green: ComputerUseTeamConsentTests 11cmux-ci step at 3cb226743cb2267411 passed
ComputerUseRemoteSetupFlowTests 7, JoinConsentTests 41, JoinConsentInterleavingTests 1cmux-ci step at 86a71c0e86a71c0e49 passed
Setup on enable (steps 1-2), red/greensteps from the setup fix (8afed68 red, 0c20b476 green)0c20b476green; ComputerUseHelperExitTests is also red on base 84ae802 (step e0d356739cbbb6561e46cc0c)
CmuxComputerUseHelper, 15 suitesstep 7a1bec592b7964c09f24ad80dd7bafbc110 passed
CmuxNext, 21 suitesstep c46e77bb52b4005096e27b473607e1d1114 passed; 1 also red on base (SettingsSchemaExportTests)
CmuxDeviceHub, 9 suitesstep dc56e4535554741ac4922a2c3607e1d134 passed
CmuxRemote (incl. end-to-end mode/client)step eecb30ec5a15674b9fba5fe93607e1d16 passed
Card layout red/greensteps d8f99ebc1418c07a43fdf7a0 / 9fc48905e8b8852ee6babe379d9b4a4bred, then green
cmux-remote-host on Linux with the real driver, clippy, Windows-gnu clippyrbx lane cu-integ-hostdd7bafbc188 passed, clean
Rust: approval crate, device hub client, acpmux bridge, CLI, daemonrbx lane cu-integ3607e1d1310 passed, clean
Device hub (TypeScript)rbx3f84c02e134 passed
iOS simulator build + Computer Use / Screen Sharing testsfleet sim step 9bbe73bee7e369bf6d03d2b36a0bf22529 passed

Live runs:

  • Windows Server 2022, Google Cloud VM, real pinned driver (54 s video). Notepad was typed into in the background through its UI Automation element; cmd was refused as a terminal. The VM is deleted.
  • Linux X11, Freestyle Ubuntu 24.04, real pinned driver (65 s video). The editor was typed into on the VM's own desktop; the terminal was refused; discovery listed apps without titles.
  • Linux X11, Freestyle VM, real pinned driver (approvals).
    • The agent was refused before the grant. After "Allow for This Session" it typed into xterm and the command ran (/tmp/cu-ok.txt = cmux-computer-use-ok). An ungranted app stayed refused.
    • Discovery listed apps and windows with no title. The planted title SECRET-xclock-window-title appeared only after the grant.
  • macOS. The real approval card, from fleet build 7b7c3eae, ran on cmux mini mini-cmux7s through the app socket.

Observed / Not run

  • Observed: everything in the table above, plus the live Linux X11 runs and the Mac card captures in the report.
  • Not run:
    • Wayland, GNOME and KDE desktops (portal grant, polkit dialog).
    • Windows Hello.
    • A live device hub (not deployed).
    • The real iPhone.
    • A live end-to-end agent flow on macOS: no driven app, no person clicking the card.
    • Setup step 3 on a real joined Mac: the macOS grants on the capture mini need a person, and the shared mini's settings file is not mine to edit. Covered by the tests above with the real JoinConsentModel.
  • Not run: the pinned-driver manifest acceptance test on macOS. It skips because no fleet worker can fetch the SDK tarball from R2. On Linux the same driver commit accepted the generated manifests live.
  • No CI coverage for cmux-tui/crates/cmux-remote-host: it is its own workspace, excluded from cmux-tui's, so no CI job builds or tests it. It ran by hand on rbx as above.

Review

Four independent review subagent rounds:

Follow-ups:

  • A Mac joined with the one-line join while labs.machineJoin is off gets consent_required, but no card and no switches. It fails safe.
  • A Turn On that fails shows its error only in Settings.

No review found a path that turns Computer Use on or approves without a person. Every finding is fixed or listed as a follow-up in the report.

Review (subagent): SHIP at c60cfe97b0a64659b95a1b0dbe6947e37869dcf6

Changelog

Added: Computer Use for agents on the Mac, Windows and Linux. Turning it on walks through the macOS permissions (and, on a team Mac, Screen sharing and Control). An optional Ask mode asks before an agent uses each app (Allow Once, Allow for This Session, Always Allow, Deny), answered on the Mac or the iPhone.

Proof

Proof report (all 26 design decisions, the security model, screenshots and video for macOS, iPhone and Linux, all builds, tests and reviews): a self-contained HTML file on the author's machine at worktrees/cmux-next-computer-use/.cmux-scratch/computer-use/computer-use-report.html in cmuxterm-hq (publishing it as an artifact needs a claude.ai login in that session).

No capture: cmux-ci capture could not lease a capture mini from this Mac (SSH key refused), and Blacksmith macOS had no runner (run 41765558711). Real captures of fleet builds on cmux mini mini-cmux7s (app socket debug.window_snapshot) are in the report.

šŸ¤– Generated with Claude Code