cmux NEXT: agents can screenshot windows, panes and tabs; browser pages come back masked

Merged
TE
7d ago
+1266-402

Summary

Agents and scripts can capture windows, panes and tabs with cmux window|pane|tab screenshot. Automation receives an image with browser chrome, tab strips, chats, saved chat frames and unchecked app pages filled opaque gray. Raw page capture remains restricted to the browser host's guarded path.

The screenshot uses the live view tree and samples mask regions across capture. It omits child/floating windows and refuses an image when masking fails or geometry does not settle. Every control-socket invocation is marked as automation, including one claiming origin: user; only an unmarked in-app user invocation without caller metadata receives raw pixels.

The existing screenshot mask from #23983 is consolidated into ScreenshotMask. Geometry/timing live in CmuxNextScreenshots; app-specific view rules remain in AppWindowImage.swift.

Review fix

Requested: “mask the strip, or mask by page-derived content instead of by one view” and “add a test where a tab title is masked in a window and a pane capture”.

TabStripView is masked independently of BrowserChromeView, covering the sibling strip in the default pane layout. The separate saved-launch-image rule remains reachable. Regression coverage verifies raw title/launch sentinels are visible, then masked in full-window and offset pane captures; a separate geometry test requires both mask regions. Crop assertions account for the pane's flipped coordinate system.

Validation

  • Clean base merge to 7d7ec329, then 5d6d95a5 fixed two fail-open review findings: unchecked native page tabs are now masked whole (shownNativePages), and an area that moves between readings now refuses the capture.
  • Display-worker fleet run (CMUX_TEST_REQUIRE_GUI=1) at 5d6d95a5: MaskedScreenshotTests 12/12 and ActionOriginTests 11/11 pass. On base, ActionOriginTests 11/11 pass.
  • Independent Astra review (high, fast): SHIP at 5d6d95a5.

Risk: high, because this changes automation screenshot privacy. Each capture owns a sampler stopped with defer; settling uses a one-shot timer, and capture retries stop after three attempts. Frames are compared within one sampler only.

Proof

No capture: this changes the returned automation image rather than app UI. The display regression captures a real window and checks raw versus masked pixels.

Changelog

Changed: agents and scripts can screenshot windows, panes and tabs with sensitive page and chat regions masked.

Agent session: agent session