feat(desktop,host): agents drive the Mac's Browser tabs over CDP

Merged
DA
11d ago
+1941-191

MD of the "Browser agent tab automation setup" rollout (plan). With this, an agent on a task can drive that task's Browser tabs on the developer's Mac, through a Mac command of kind browser. It only works while both of these are on: the Mac's "Let agents run commands on this Mac" switch (macCommands, Dean's call on 2026-10-20) and the Browser tab experiment. The service's routing of the browser tool to Mac tabs is ME's job and is not in this PR.

What changed

packages/features

  • macBrowserParamsSchema / MacBrowserParams in @luke/features/mac-command-wire: { taskId: uuid, actions: browserAgentActionsSchema }. ME queues these as the command's params. There is also an isMacBrowserCommand guard.
  • B2's CDP runner moves out of the service (git mv) into @luke/features/browser-agent-cdp, along with its snapshot formatter (browser-agent-snapshot.ts) and the session types (@luke/features/browser-cdp: CdpSession, CdpFailure, …). The Mac and the cloud now run the same runBrowserActions, so the vocabulary and the results match by construction.
    • The service's suites are unchanged apart from their import paths, and still test the runner against the fake Chromium.
    • The one addition is a CDP_FAILURE.DENIED kind, whose detail is the reason the agent reads. A refusal's words don't get rewritten as "there is no tab …", so the Mac uses it for Stop.
  • BrowserPage.agent?: boolean. browser.open now carries taskId. There is a new BrowserOpened push payload.

packages/host

  • compose-mac-commands.ts (now on main's live follow, #3285):
    • every follow and every presence beat names browser only while the Browser tab experiment is on; when the experiment changes, the follow ends and the next one names the kinds that run now;
    • a browser command whose params don't decode is refused in words;
    • a run is held to the command's timeoutSeconds, and is cut off within a second once macCommands, the sign-in, or the experiment goes off, answering "Stopped: … switched off";
    • the result is posted through settleMacCommandResult, as MC set up.
  • A new node capability, browser.act (node-capabilities.ts). client/gateway-wiring.ts perform decodes the params and hands them to the node's browserAct. A client without one, such as the web tab, answers unavailable.
  • mac-browser-tabs.ts, the logic:
    • It speaks to one task's tabs as one CdpSession, each tab through its own view's webContents.debugger. A view's debugger reaches only its own page, so the few browser-level commands the runner sends (attach, detach, activate, close) are answered here over the task's tabs.
    • macBrowserTabsOver decides not-run: experiment off, window closed, or a task the window neither shows nor holds a tab on.
    • runMacBrowserCommand builds the MacBrowserResult.

apps/desktop

  • browser-views.ts keeps each view with the task its tab was opened on.
  • agentTabs(panel, taskId):
    • lists every tab on the task, whichever task the window shows;
    • open creates a tab on the task, gives it a default 1280×800 size until it's placed, and pushes app:browser-opened to the panel, which draws it through the "+"'s path (use-side-panel.ts, withBrowserChosen) without moving keyboard focus;
    • close pushes app:browser-closed.
  • While an agent drives a page, its debugger is attached and background throttling is off. Both go back to normal when the last command lets go.
  • Screenshots: Page.captureScreenshot is intercepted and served by capturePage(…, { stayHidden: true }), resized to the clip's scale and encoded at the runner's quality. This means a hidden or never-placed view can still be captured.
  • A new act, browser.stop. Stop detaches the debugger at once and refuses every later agent action on that tab until the page gets focus again, meaning the developer clicked into it.

packages/panel

  • browser-pane.tsx: while agent is true, an @luke/ui Banner strip under the bar says "An agent is using this tab", with a Stop agent button. It's named so it doesn't clash with the bar's own Stop step while a page loads.
  • useAgentBrowserTabs (browser-pane.tsx), called from use-tasks-tab.ts, listens for the two new pushes while the experiment is on, whether or not the side panel is drawn. When an agent closes the chosen tab, the tab viewed before it is chosen, as a tab's own close does.

The result ME reads

  • params: macBrowserParamsSchema from @luke/features/mac-command-wire.
  • output: a BrowserAgentRun (browserAgentRunSchema from @luke/features/browser-agent-actions), the same shape as a cloud run, with two differences:
    • each screenshot step keeps mime/width/height but its image.data is "";
    • image is the last screenshot of the run, as JPEG base64.
  • To stay under MAX_BROWSER_OUTPUT_CHARS, the text of the earliest snapshot and console steps is dropped first (text: "" or messages: [], with truncated: true), so the latest snapshot arrives whole. If that isn't enough, each list's addresses and titles are cut to 200 characters. A run that is still too long is answered as a refusal in words.
  • Tab ids are the ids the panel made, as list and open return them.
  • not-run comes back as { refused: "Not run: …" }, in words the agent can act on.

Decisions

  • Screenshot size follows B2, not MC. The runner's own BROWSER_SCREENSHOT limits apply (longest side 1280, JPEG, ≤180 KB, qualities 70/50/30), so a Mac screenshot is the same size as a cloud one. MC's macScreenshotEncoding (1568 px, quality 80) disagrees; it's left as it was and isn't used.
  • "The window is closed" means no panel window, or a panel window that isn't visible. Closing Luke's window only hides it.
  • "Shows the task" means app state's tasks.activeTaskId. A task with a tab already open is reachable even while the window shows another task.

Privacy

This stays within the line amended in #4257. The address, title, snapshot and screenshot reach only the command's result, which goes to the service. Nothing new goes into a trace, a counted event, a fixture, app state, or the voice window:

  • the panel's Browser tab state is never stored;
  • the node invocation is no event;
  • reports name no page.

Tests

  • packages/host/src/mac-browser-tabs.test.ts (fake tabs and debuggers):
    • refusals in words for experiment off, window closed, and task not shown;
    • reach while the task is shown, or while a tab is held on it;
    • actions land on the named tab and never on another task's;
    • open followed by later actions in the new tab;
    • screenshot scaled to 1280 on a Retina page, falling back to the next quality to get under the byte cap, and sent as the image;
    • Stop refuses actions, both before a run and mid-run;
    • a run of long snapshots stays under the output cap.
  • compose-mac-commands.test.ts: browser is advertised only with the experiment on, and the switch gates everything; unreadable params are refused; a good command runs and its result is posted.
  • browser-acts.test.ts: Stop reaches only the panel's own page. browser-pane.test.ts: the strip and its Stop. use-side-panel.test.ts: an agent's tab is chosen on the open plan, only held on another, and goes when closed.

Not checked on a Mac

CI runs on Linux only, and I couldn't run Electron against a real page here. These are untested:

  • webContents.debugger input on a real page;
  • capturePage with stayHidden on a hidden or never-placed WebContentsView;
  • whether a view an agent opens on a task the window isn't showing (one that has never been added to a window) loads and lays out at its default size;
  • whether the focus event (which lifts Stop) fires only on the developer's click and never on CDP input;
  • that the agent's tab is drawn without moving keyboard focus.

These belong in the plan's manual Mac pass (step 12).

Made with Luke

Devin Review