#4380 feat(service,features): the agent's cursor moves on a cloud Browser tab

Part 1 of 2 of the agent's cursor on a cloud Browser tab. Dean chose the cursor only, for cloud tabs, so nothing here touches Mac tabs. This PR gets each cursor move from the agent's actions to a read the panel can follow live. Part 2, stacked on this one, adds the client follow and the overlay.

What changed

The runner (@luke/features/browser-agent-cdp)

• runBrowserActions(cdp, actions, { cursor }) takes an optional BrowserCursor<R> = { report(move), watched }.
• Click, type and a wheel scroll each call report with the resolved point just before their input is dispatched:
  • click: the given point, or a ref's point;
  • type: the ref's point, or the focused element's middle;
  • scroll: the middle of the page.
• When watched is true, the input then waits BROWSER_CURSOR_BEAT (250 ms), so the glide lands with the action.
• A caller that passes no cursor sends exactly the same commands as before. That covers the Mac path and settleTitles.
• Fetching a point or the page size never fails the action. If either can't be read, no move is reported and the action runs as before.

The vocabulary

• BROWSER_CURSOR_KIND (move | click | type) and BrowserCursorMove { tab, x, y, kind, viewport: { width, height }, at } live in browser-agent-actions.ts.

The store

• An additive, idempotent migration adds nullable columns to agent_browser_tab: cursor_x, cursor_y (double), cursor_kind, cursor_viewport_w, cursor_viewport_h (int), and cursor_at.
• pointBrowserTab(conversationId, move) is one UPDATE keyed by conversation and target, with no read first. It also moves last_acted_at on.
• A tab that has no row yet (one this same call opened) takes no move. No window can be drawing it yet.

The browser tool

• It passes a cursor that writes each move to the tab's row: at most one write per action.
• Who counts as watching: the beat applies only while the agent's sandbox_terminal_lease stands. The live view moves that lease on while a panel has one of the agent's tabs open, with an idle bound of about 5 minutes.
  • It is read once per call. An agent nobody watches is never slowed.
  • The rule is coarse: a Terminal tab in use holds the same lease, so it counts as watching too.
• Failed store writes are logged by their kind only. The point and the kind are never logged. They are not page content, and they fall under AGENTS.md's cloud-browser line.

The wire and the held list

• Each AgentBrowserTab now carries an optional cursor, present once every cursor column is written.
• agentBrowserTabLineSchema and agentBrowserTabOf(line, now) are now the one place a tab is built. The held list and the live read both go through them, so the two can't disagree.

The live read

• GET /api/live/task-browser-tabs?id=&userId= is live.taskBrowserTabs in the contract.
• It is one sealed statement, taskBrowserTabsLiveStatement in @luke/database/models/agent-browser-tabs.
• It enforces the same reach rule as the held list:
  • a member join on the task's workspace;
  • taskSeenBy;
  • agentReachableBy;
  • an inner join to the coding agent.
• Polling, held after polls, and a fallback when sealing fails all come from sealedParts.

What part 2 needs

• Contract: hosted.live.taskBrowserTabs, input { id: taskId, userId }, path HOSTED_SERVICE_PATH.LIVE_TASK_BROWSER_TABS.
• Answer: taskBrowserTabsLiveAnswerSchema = { url, queries: { tabs: sealedLiveQuerySchema } } (@luke/features/live-wire).
• Rows: taskBrowserTabsLiveRowsSchema = { tabs: AgentBrowserTabLine[] }, where each line is:
  { id, agentId, address, title, openedBy, openedByUserId, driverUserId, heldUntil, createdAt, lastActedAt, cursorX, cursorY, cursorKind, cursorViewportW, cursorViewportH, cursorAt }
  Instants are epoch ms, and every column is nullable except id, agentId, address, title, openedBy, createdAt and lastActedAt.
• Reader:
  • browserTabsOfLiveRows(rows, now) returns AgentBrowserTab[], sorted the way the held list is (createdAt, then id). Its result equals the held list's.
  • nextBrowserTabLapse(rows, now) gives the instant to read the rows again. A lapsing takeover changes no row, and a live statement can't read the clock.
• Cursor on a tab: tab.cursor?: { x, y, kind: "move" | "click" | "type", viewport: { width, height }, at }.
  • x and y are CSS pixels of the page's visible area; scale them by viewport onto the drawn live view.
  • at is epoch ms, when the input was about to land.
  • Rocking or idling can be judged from cursor.at or lastActedAt.
• Client follow (out of scope here): the gateway method and event, the compose-tasks follow (copy followPages), the host-operator and act-port procedure, swapping useCloudBrowserTabs from its 5 s poll to the follow, and the lapse timer.

Tests

All tests are hermetic: PGlite, a fake CDP peer, and TestClock.

• Runner:
  • the cursor hears each click, type and wheel point, with the page size, before its input;
  • typing where the page can't point to moves no cursor;
  • a watched click waits the beat (checked by mutation: the test fails without the sleep);
  • the existing exact-command tests pin that runs without a cursor are unchanged.
• Tool:
  • a click writes the cursor to its row, and a tab this call opened takes none;
  • the beat applies only while the sandbox lease stands.
• Store: pointBrowserTab writes only to its own agent's tab.
• Wire: the held list carries the cursor once the agent points.
• Live:
  • the live rows equal the held list for the starter, a teammate and a reader, through a point, a takeover and its lapse, and an outsider gets nothing;
  • the read is sealed for the bearer and the task, a stranger gets not-found, and a poll returns the rows inline;
  • the statement is in live-statements.test.ts, which checks it is immutable and seals through Neon's adapter.

main is merged in after #4374 (browser waits), which also edited the runner. Both changes are kept: the navigation watch runs before the action, and the cursor report and beat run inside it, before the input.

Made with Luke

---