fix(desktop): make computer use act on the app the user sees
Merged
WA
24d ago+258+258+012345678901234567890123456789-56-56-01234567890123456789
Summary
Computer use failed many tasks because the native helper refused, or misreported, actions that worked. Measured on MacAgentBench single-app tasks (Opus 5.5 through the worker loop, fresh macOS VM per episode), this change raises paired success from 37/83 to 53/83 (exact McNemar p = 0.0001). It also cuts tool calls per task from 21 to 12.6, cost by 34%, wall time from 101 s to 65 s, and failed tool calls from 4.6 to 0.3 per task. On the tasks the VM image can solve, three runs averaged 79.1% (sd 2.6) against 53.7% before.
- Coordinate input: the occlusion check took the topmost window-list entry. On macOS that is Notification Center's or the Dock's full-screen click-through window, so every click, scroll and drag failed with "covered by another window". The target is now the app that Accessibility hit-testing routes the point to, and input is refused when hit-testing gives no answer, so the app's own sheets, popovers and menus count as the target.
- AX presses: a press that opens a sheet or modal holds the app's reply past the 0.25 s timeout, and
cannotCompletewas reported as a failure after the action had happened. Actions now use a 2 s timeout and treatcannotCompleteas dispatched. - Snapshots: a request that fails before anything reaches the app keeps its snapshot, in both the helper and the service, and reports
dispatchState: "not_started". Previously every refusal forced a re-observation and a retry. - Typing and keys:
type_text/input_sequenceaccept any focusable element, including Terminal's read-only text area, combo boxes and editors whose focus sits on a child.press_keygoes to the app's focused window, which includes sheets and popovers. Common key aliases such as Esc, ArrowLeft and PgUp are accepted. - Menus: open menus and their submenus are observed first, and menu item labels include their shortcuts (⇧⌘N). Some apps (Calendar on macOS 26) answer AXPress on a menu item by highlighting it without running it. When that leaves the menu open, the helper completes the press with Return.
- Schema:
press_keymay omitwindowId(the key goes to the focused observed window, or a dialog above it), and mutations acceptincludeScreenshotas shorthand forobserveAfter.includeScreenshot. Models sent both shapes often, and both were rejected before reaching the app. This mirrors a worker contract change, which should merge first so the contract sync stays in step. - Projection budget: Mothership stores a computer result over 24,000 characters and shows a preview cut through the middle of the tree. The projection now fits under that limit (22 KiB), so its prioritized rows survive.
Test plan
- Benchmark A/B on 83 paired tasks, plus two more runs on the solvable set (numbers above)
- Manual checks in a macOS 26 VM: coordinate double-click in Calendar, Notes sheet button press, typing into Terminal, Calendar New Calendar > On My Mac, snapshot reuse after a refused action
- Service regression tests: a refused action keeps the snapshot; an action with unknown dispatch consumes it
- CI