#27030 acpmux: peers recover right after a sleep; an unforced peer_reconnect leaves an open link alone

acpmux peers recover right after this computer sleeps, and an unforced _acpmux/peer_reconnect leaves an open link alone.

Leo, 2026-10-28: "after you've connected something remotely and whatever, that whole setup situation, like after we set it up, how good is it in terms of that whole, like, recovery flow, and, like, that kind of connection situation, and how reliable things are?"

Leo, 2026-10-28: "whatever's convenient for cloud, like that can be productionized, operationalized, we ought to just put out there"

Walkthrough and gaps: manaflow-ai/cmuxterm-hq#4877. This covers gaps G2 and G3 there.

What changed

• A sleep is noticed (peer_link.rs SleepWatch).
  • The problem: a link's timers run on the monotonic clock, which stands still while the computer sleeps. After a wake, a dead ssh tunnel read connected until ssh's keepalive (about 45 s) or the 80 s silence budget noticed. It then waited out up to 60 s of backoff.
  • The fix: every 5 s the link compares the wall clock with the monotonic clock. A gap of 10 s or more is a sleep.
• An open link after a sleep (peer.rs connect_once): it sends a ping at once. If nothing answers within 10 s (WAKE_PROBE), the link drops and redials. A live link answers, and nothing changes.
  • The clocks are also read before every write to the link (the regular ping and each request), so a sleep is noticed even when the ping's tick is served first.
  • While that probe runs, a regular ping or a request written to the link gives up at the probe's deadline (write_deadline), not after the 80 s send budget, so a write to a tunnel that died in the sleep never holds the probe up.
• A backoff wait the computer slept through (peer.rs wait_backoff): it ends at the wake, and the backoff starts over at 2 s.
• _acpmux/peer_reconnect takes force (hub/peers.rs reconnect_peer_with).
  • Without force, a peer whose link is open (connected, or degraded: open but slow, or in a handshake younger than its own 90 s deadline) is left alone. Before, the app's reconnect on a send to a slow machine replaced the peer, which cut the working link and failed the requests on it as peer_disconnected.
  • With wait: true (the app's pick and send), a handshake still in progress is waited for, up to the same 30 s a new attempt gets, so the caller never sends to a machine that is not connected yet.
  • host reconnect and host doctor --fix pass force: true, so the CLI behaves as before.
  • Hub::reconnect_peer (the Rust API the tests and pairing use) still forces.

Observers and timers (stop condition and cleanup)

• The link's wake check:
  • one 5 s tokio::time::interval per open link, plus at most one 10 s probe deadline;
  • both are locals of connect_once, so they end with the link (any exit of its loop);
  • each check is two clock reads.
• The backoff wait: it wakes every 5 s to check the clocks and ends at its deadline, on a wake, or on stop. It replaces the single sleep it had.

Risk

Low to medium, contained to acpmux peer links.

• A wall clock stepped forward by 10 s or more (an NTP correction) reads as a sleep. That costs one early ping (a live link answers it) or one early attempt.
• A clock stepped backward is no sleep.
• An app that sent peer_reconnect to drop a connected link now needs force: true. Only the CLI did that; the pane's reconnect is meant for down machines.

Tests

On Linux CI (cmux-ci test):

• cargo test -p acpmux --lib peer_link, including:
  • a_sleep_is_the_wall_clock_running_ahead_of_the_monotonic_one;
  • a_wall_clock_set_back_is_no_sleep.
• cargo test -p acpmux --lib peer_link also covers a_write_during_a_wake_probe_gives_up_at_the_probes_deadline.
• cargo test -p acpmux --test peer_state_events:
  • a new step: an unforced reconnect of a connected peer sends no state note and leaves it connected;
  • a new test: a peer holds its initialize answer, and an unforced wait reconnect waits for that handshake and leaves the link it waited for up (the peer takes one connection, so a replaced link would never connect).
• cargo test -p acpmux --lib host_peering, --lib peer:: and --test peer_lease.
• cargo clippy -p acpmux --all-targets -- -D warnings: clean.

Not run: a real lid-close on a Mac. The wake path is the clock comparison above, plus the existing ping and redial.

Agent session: agent session

🤖 Generated with Claude Code