feat(service,dev,luke): the browser tool

Merged
DA
5d ago
+1672-55

PR D of the "Browser agent tab automation setup" rollout. It gives Luke (the brain), coding agents, and both their workers one agent tool, browser, which drives the Chromium in the agent's own sandbox (B1, B2) and keeps the agent's tabs in the service (C).

What it does

  • The tool: AGENT_TOOL.BROWSER, with its roles row for BRAIN, BRAIN_WORKER, CODER and CODER_WORKER. Its input is { actions: browserAgentActionsSchema }, emitted through the shared JSON Schema emitter.
  • One routing point: packages/service/src/agents/browser/browser-tool.ts. Its browserFor is the only place a call is routed to a browser, so ME adds Mac tabs there.
    • The browser session is browserSessionOf(session), so a worker drives its root's browser.
    • The tabs belong to the admitted conversation: a coding agent's own, the task's for Luke, a worker's parent's.
  • Takeover: once the browser is reached, the call reads the takeovers (browserTabDriver). The first action on a tab a member drives answers a failed step that says so ("…wait until they hand it back… or ask them"), and nothing after it runs. The steps before it stand.
  • Tab store:
    • A tab that is opened or acted on is upserted with its current address and title.
    • close drops the tab.
    • list makes the store match the browser's own pages.
    • The result also carries tabs: the tabs the call acted on, as they now stand.
  • Unreachable browser: answers not-run with Vercel's reason, never a throw.
  • Tab restore (restore-tabs.ts, added at the orchestrator's request):
    • When AgentBrowser.ensure reports started: true (a resumed sandbox), restoreBrowserTabs reopens every kept tab at its last address.
    • It moves the row onto the new target with the new retargetBrowserTab. The tab keeps its id, its opener, and any takeover.
    • connectRestoring is exported for E's relay (#4314) to use as a follow-up.
  • Screenshots:
    • They reach the model as image parts (browserModelOutput, chosen in the brain's tasksModelOutput and in the dev tool's toModelOutput).
    • The stored conversation keeps each step without its picture: the relay's storedLook calls storedBrowserOutput, as it does for look_at_picture.
    • No page content is logged or traced; a store failure is logged by its kind only.
  • Wiring:
    • The brain's BRAIN_TOOLS and BrainCall.browser; the brain worker gets it through the registry.
    • apps/dev/tools/browser.ts and apps/dev/subagents/worker/tools/browser.ts, through browserFor(admit) in apps/dev/host.ts (the same pattern as runOnMacFor).
    • context-estimate.ts counts the tool.
    • Each eve root's runtime (apps/luke/runtime.ts, apps/dev/runtime.ts) now provides AgentBrowser (agentBrowserServices = agentBrowserLayer over sandboxPtyLayer) on top of webServices. The prompt named host.ts, but the roots' layer is built in runtime.ts, so it's composed there.
  • Instructions:
    • Coding agents get a "Your browser" section: use it for what the developer should see or help with (a flow to show, a preview, a sign-in, MFA, a CAPTCHA); keep throwaway checks on sandbox Playwright; the developer may take a tab over, and the agent waits until it's handed back.
    • The brain's tool list and both workers' prompts get a line each.
  • Transcript row: CALL_KIND.BROWSER. The line reads like "Clicked, typed in Sign in · Acme", using the tab's title from the answer, or the page it went to while the call runs. It is tagged "Browser" in the place "On your Mac" takes, with the Browser tab's globe icon. Words are masked as every other row's are.

Golden

packages/service/fixtures/coder-instructions/chat-off.json was rewritten with LUKE_UPDATE_FIXTURES=1. The only change is the new "## Your browser" section, in each brief's on-task and taskless prompt.

Tests

  • browser-tool.test.ts, over fakeCdpPeer and the real store on PGlite:
    • open and navigate keep the tab with its page; close drops it;
    • list reconciles the store with the browser;
    • an action on a driven tab is refused and nothing after it runs, then it runs once the takeover lapses;
    • a fresh browser reopens kept tabs as the same tabs, takeover included;
    • the screenshot is in the model output and not in the stored output;
    • an unreachable browser answers why.
  • coder-host.test.ts: a relayed browser call's transcript keeps its steps and never the picture. This test fails without the relay wiring.
  • brain-host-tools.test.ts: Luke's call through runTool keeps the tab under the task, listed with agent: null.
  • coder-tool-files.test.ts: the worker's tool files must equal toolsOf(CODER_WORKER), like the agent's own. This stands in for a "tool set lists browser" test, which would be a change-detector test.
  • agent-browser-tabs.test.ts: retargetBrowserTab.
  • tool-call-model.test.ts: the browser row.

Not in this PR

  • Claude Code and Codex agents reach Luke's tools through the Luke MCP server, which doesn't offer browser yet.
  • The live view and takeover relay (E), the cloud Browser tab (F), and Mac tabs (MD, ME).

Made with Luke

Devin Review