cmux NEXT screen sharing host: modified clicks, button drags and pixel scrolling

AZ
8d ago
+433-18

Summary

Controlling a Mac from the cmux iPhone app had three input gaps on the host side (ScreenCaptureBackend's EventInjector):

  • Modified clicks didn't work. The viewer sends a modifier key down, the click, then the key up, but mouse events carried no flags, so ⌘-click, ⇧-click and ⌥-click arrived as plain clicks. Every mouse event (move, drag, down, up, scroll) now carries the modifiers the viewer holds, until their key up. A chord no longer releases a modifier the viewer was already holding (sticky ⌘, then ⌘C, then a click is still a ⌘-click).
  • Middle drags were plain moves, right drags named the left button. A move with a button held is now that button's drag, picking left, then right, then middle (.leftMouseDragged / .rightMouseDragged / .otherMouseDragged), with that button's number. A scroll during a drag keeps the drag instead of posting a plain move first.
  • Scrolling was whole lines only, which feels coarse from a phone. This adds pixel scrolling as a backward-compatible protocol addition (agreed with the iOS viewer work):
    • When the input data channel opens, the host sends {"t":"caps","s":"scroll-px,mods"} (comma list) before its first displays message. Only ScreenCaptureBackend advertises capabilities (new ScreenBackend.inputCapabilities, default none), so the Screen Sharing loopback backend sends nothing; a view-only session sends nothing.
    • A viewer that saw scroll-px may send {"t":"scroll","x":..,"y":..,"dx":<px>,"dy":<px>,"u":"px"}. The host posts it as .pixel (continuous) wheel events with the same sign as a line scroll in that direction (dy is wheel 1, dx wheel 2). u is a new optional field on ScreenInput, nil by default and omitted when encoding; without it the scroll stays in lines. Fractional dx/dy round to the nearest pixel instead of failing the whole message.
    • ScreenStreamViewer routes caps to a new onCapabilities callback and only displays messages to onDisplays. An older viewer reads any host message as displays; it gets caps first and the real displays right after, so it ends with the right list.

View-only sessions still drop all of this input in ScreenStreamHost.deliver (test added).

The change stays inside the input handling of ScreenCaptureBackend.swift and the host data-channel code in ScreenStream.swift, so it moves as-is into the daemon-owned host process being designed to run Computer Use and Screen Sharing together.

Testing

EventInjector takes an injectable poster, so the tests capture the CGEvents the host would post (no Accessibility or Screen Recording needed). New ScreenInputInjectionTests and ScreenInputWireTests in Packages/Shared/CmuxVNCWebRTC/Tests/CmuxVNCWebRTCTests/ScreenInputInjectionTests.swift.

Command (fleet): cmux-ci run --class light --script scripts/ci/package-test-lane.sh --ref <sha> --arg=suite --arg=Packages/Shared/CmuxVNCWebRTC --arg=<suites> --env CMUX_CI_XCODE_APP=/Applications/Xcode_26.6.app

RESULTS_PLACEHOLDER

Observed / Not run

  • Observed: the fleet steps above.
  • Not run: live verification with the iPhone viewer (modified clicks, middle drags and pixel scrolling on a real Mac host). The coordinator does that with the viewer build. The tests assert the events built, not how apps respond to them after the HID tap.

Changelog

Fixed: ⌘-, ⇧- and ⌥-clicks and middle-button drags from the cmux iPhone app work on the shared Mac
Added: Smooth pixel scrolling when controlling a Mac from the cmux iPhone app

Proof

No capture: host input handling with no UI; covered by the event-capture tests, live phone check by the coordinator.

🤖 Generated with Claude Code