cmux-rd host 0.2.2: the person master exists only while the person acts, so the agent's keys reach Chrome

Merged
TE
1d ago
+518-171

What

On a Cloud desktop, the agent's typing into Chrome works again 3 s after the person stops using Take over. cmux-rd host (cmux-rd 0.2.2) now keeps the person's XI2 master cmux person only while the person is acting.

Leo asked for this:

is that really necessary? can't you both do it at the same time?

Also go check in on the state of affairs of screen sharing remotely on these machines, like the cloud machines, because there was so much VNC work. But it's also like, I feel like it wasn't really linked up end to end here, and we have an opportunity quite soon to start doing that. [...] And then it became AI computer use!!

Why

Chromium reads key events from only one master keyboard.

  • ui/events/devices/x11/touch_factory_x11.cc sets virtual_core_keyboard_device_ to each master keyboard it sees, so the last one listed wins.
  • ShouldProcessDeviceEvent drops key events from every other device.

So while cmux person exists, every key the agent sends into Chrome is lost, even when the master sits untouched.

Staging run 42083: keys landed with no master present and were lost once the master existed. They worked again right after it was removed.

Changes (cmux-tui/crates/cmux-rd-host)

On-demand master

  • The person's next input creates the master again. The host then moves the pointer back to where the person left it.
  • The master is removed when either of these happens:
    • 3 s pass (PERSON_IDLE) with no input, no key or button held, and nothing queued.
    • Control ends.
  • Desktop::pump now returns the next wake time instead of a bool. The session loop sleeps until then, so it doesn't spin.

Marker matches the master

  • The root property _CMUX_RD_PERSON exists exactly while the master exists. It is written when the master is created and deleted together with it.
  • button_held also covers a press that is still waiting behind the agent's held button.
  • Keys now refresh the property, just as buttons and scrolls do.
  • When control ends, the session loop's release_all releases held input, drops the queue and removes the master and the marker.

Cross-process coordination (Codex F1/F2)

  • Hosts that use the master list their pids in _CMUX_RD_PERSON_HOSTS. Each host changes that list under a server grab.
  • If a live host (a running pid with the host's process name) still lists the master, a new host reuses it and never removes it.
  • Only the last live host removes the master.
  • If the master exists but no live host lists it, it came from a host that crashed. The new host removes it, which releases whatever it held, and creates a fresh one.

Per-press hold deadline (Codex F2 on #36614)

  • Each queued event records when it arrived.
  • A burst of presses queued behind the agent's held button no longer waits 2 s per press. Each press waits at most 2 s from its own arrival.

Nits

  • The systemd.rs header now matches the README: socket root:cmux 0660 in a 0711 dir, and the link receiver runs as uid 1000.
  • The test Xvfb guard checks Child::try_wait instead of /proc.

The version goes to 0.2.2 in the crate's Cargo.toml, the crate's Cargo.lock and the cmux-tui lock.

Tests

Fleet run 42093900919 on Linux: fmt, clippy -D warnings, the whole crate with --include-ignored (34 passed; the inject tests then 3 more times), and the cmux-tui lock with --locked.

New tests:

  • hold_guard_counts_each_press_from_when_it_arrived (unit test).
  • person_master_goes_when_idle_and_comes_back_where_the_person_left (Xvfb):
    • No marker exists before the first input.
    • The master and marker appear together.
    • A held key keeps the master past the idle limit.
    • After the idle limit, both go.
    • The next key creates the master again with the pointer at (700, 500).
    • release_all removes both.
  • marker_shows_a_press_waiting_behind_the_agent (Xvfb).
  • person_master_another_live_host_uses_is_kept (Xvfb):
    • Another live host's master is reused, and its button stays down.
    • Its pid stays listed and the master is not removed.
    • After that host dies, the next host replaces the master and nothing stays held.

The crash-reuse test from #36614 still passes.

Staging Chrome run

Staging run 42092881189 used two Chrome windows on :99 with no window manager. It walked the 0.2.2 lifecycle:

stepresult
agent types, no masterlands
person takes over: master and marker created, person clicks and typeslands
agent types while the master existslost (the Chromium rule above)
3 s idle: master and marker removed
agent typeslands
person returns: master created again, pointer restored to (930, 536), person typeslands
agent types while the master existslost
idle again, agent types twiceboth land

The companion cmux-cua change makes the agent wait, bounded, while _CMUX_RD_PERSON exists before it types into Chromium. A press waits only when the person is using that window. That change is in manaflow-ai/cmux-cua.

Known limits (review follow-ups, not needed with one host per machine)

A Cloud machine runs one cmux-rd host. Codex's two follow-ups only apply when two hosts share one display at the same time:

  • F1: one host's release_all sends releases on the shared master. It can release a key or button that another live host still holds.
  • F2: a host can die between the liveness check and the reuse. The next host would then keep that host's held input instead of replacing the master.

Changelog

  • Cloud desktop: after you take over and then stop for 3 s, the agent's typing into Chrome works again.

Agent session: agent session

šŸ¤– Generated with Claude Code