docs(browser): document the screenshot auto-attach escape hatch

Merged
AL
4mo ago
+15-0

Problem

browser_execute attaches every Page.captureScreenshot result to the next assistant turn. That default is right — it saves a read call almost every time the agent screenshots something it wants to look at, and we don't want to change it.

But there was no way to capture pixels the agent does not need to see. Two real cases:

  • Bulk capture (e.g. screenshotting many pages for a dataset) attaches every frame to context, with no opt-out.
  • An artifact that only gets written to disk — an og:image card, a thumbnail — is attached even though the agent never looks at it.

Both are rare, which is why this is documentation rather than tool surface. Alternatives considered and rejected in #122: a synthetic attachToContext param on Page.captureScreenshot (teaches the agent a CDP field Chrome has never heard of, and carried the opt-out via WeakSet object identity, which fails silently in the leak direction), and a fourth browser_execute tool parameter (a permanent, eagerly-loaded schema field on every call for a rare need).

Why a documented route works

The screenshot tap subscribes via session.onCallResult(...), which fires only from _call's resolve path (session.ts:252-256) — the command/response channel. CDP events route through a separate listener list (session.ts:273-277) and never reach it.

So pixels delivered as the Page.screencastFrame event are structurally invisible to the tap. No code change, and the agent learns a genuine protocol capability that transfers to any CDP client instead of a BrowserCode-only fiction.

Change

15 added lines in skills/browser-execute/SKILL.md, next to the auto-attach explanation it qualifies. The lead sentence states the rule (response attaches, event does not) so the agent can reason about cases beyond the example.

Validation

Verified against Chrome 150.0.7871.187 with --headless=new — the harder case for compositing — via browser_execute against a throwaway debug-port profile:

  • Static, already-loaded, idle page: startScreencast delivers a frame in 7ms with no navigation, scroll, or re-layout nudge. This was the open question; the documented snippet needs no nudge and no polling loop.
  • Screencast frames produce zero attachments, across every run. Page.captureScreenshot still attaches (control case rendered inline as expected).
  • setDeviceMetricsOverride({ width: 1200, height: 630, deviceScaleFactor: 1 }) yields a PNG whose header reads exactly 1200x630, confirming the viewport is what sets frame size.
  • bun test test/skills.test.ts — 2 pass (skill still materializes with {{SKILLS_DIR}} substituted).
  • Pre-push filtered typecheck passed.

Emulation.setDeviceMetricsOverride is marked Optional: in the snippet on purpose: it reflows the page, so it is wrong as an unconditional step for bulk capture where you want the page as-is.

Protocol details found while testing, deliberately left out of the skill text

Recorded here rather than spending tokens on them in a lazily-loaded doc:

  • screencastFrameAck is not needed for navigation-paced capture — 3 navigations produced 3 frames with zero acks. It is required for continuous capture: on an animating page a 3s window yielded 3 frames unacked vs 178 acked, so Chrome stalls after a couple of unacked frames. The bulk-capture case that motivated this PR is navigation-paced, so the skill omits ack entirely.
  • sessionId on the frame event is not a frame counter, despite the protocol describing it as "Frame number". It is constant within a cast and increments per startScreencast call. Anything acking per-frame by treating it as a frame identity will behave confusingly.
  • Page.screencastFrame is flagged "experimental": true in the protocol. It is the mechanism cloud's own live preview uses, so it is stable in practice.

Zone

Green — packages/bcode-browser/ only, docs-only, no upstream diff, no EXCEPTIONS.md entry needed.

Closes #122.

Summary by cubic

Documented how to opt out of screenshot auto-attach in browser_execute. Use Page.screencastFrame instead of Page.captureScreenshot to capture pixels without attaching (responses attach; events do not), useful for bulk capture and disk-only artifacts.