fix(desktop): make computer use act on the app the user sees

Merged
WA
24d ago
+258-56

Summary

Computer use failed many tasks because the native helper refused, or misreported, actions that worked. Measured on MacAgentBench single-app tasks (Opus 5.5 through the worker loop, fresh macOS VM per episode), this change raises paired success from 37/83 to 53/83 (exact McNemar p = 0.0001). It also cuts tool calls per task from 21 to 12.6, cost by 34%, wall time from 101 s to 65 s, and failed tool calls from 4.6 to 0.3 per task. On the tasks the VM image can solve, three runs averaged 79.1% (sd 2.6) against 53.7% before.

  • Coordinate input: the occlusion check took the topmost window-list entry. On macOS that is Notification Center's or the Dock's full-screen click-through window, so every click, scroll and drag failed with "covered by another window". The target is now the app that Accessibility hit-testing routes the point to, and input is refused when hit-testing gives no answer, so the app's own sheets, popovers and menus count as the target.
  • AX presses: a press that opens a sheet or modal holds the app's reply past the 0.25 s timeout, and cannotComplete was reported as a failure after the action had happened. Actions now use a 2 s timeout and treat cannotComplete as dispatched.
  • Snapshots: a request that fails before anything reaches the app keeps its snapshot, in both the helper and the service, and reports dispatchState: "not_started". Previously every refusal forced a re-observation and a retry.
  • Typing and keys: type_text/input_sequence accept any focusable element, including Terminal's read-only text area, combo boxes and editors whose focus sits on a child. press_key goes to the app's focused window, which includes sheets and popovers. Common key aliases such as Esc, ArrowLeft and PgUp are accepted.
  • Menus: open menus and their submenus are observed first, and menu item labels include their shortcuts (⇧⌘N). Some apps (Calendar on macOS 26) answer AXPress on a menu item by highlighting it without running it. When that leaves the menu open, the helper completes the press with Return.
  • Schema: press_key may omit windowId (the key goes to the focused observed window, or a dialog above it), and mutations accept includeScreenshot as shorthand for observeAfter.includeScreenshot. Models sent both shapes often, and both were rejected before reaching the app. This mirrors a worker contract change, which should merge first so the contract sync stays in step.
  • Projection budget: Mothership stores a computer result over 24,000 characters and shows a preview cut through the middle of the tree. The projection now fits under that limit (22 KiB), so its prioritized rows survive.

Test plan

  • Benchmark A/B on 83 paired tasks, plus two more runs on the solvable set (numbers above)
  • Manual checks in a macOS 26 VM: coordinate double-click in Calendar, Notes sheet button press, typing into Terminal, Calendar New Calendar > On My Mac, snapshot reuse after a refused action
  • Service regression tests: a refused action keeps the snapshot; an action with unknown dispatch consumes it
  • CI