acpmux: _acpmux/peer_state_changed pushes each peer link state change to watchers (peerStateEvents)

Merged
TE
8d ago
+671-12

Leo's ask (manaflow-ai/cmuxterm-hq#4770, plan row 3, one machine-state feed for the composer strip and the sidebar):

"you can just not choose a project ... you just get a little green dot ... as if you were working on your own computer, but now it's remote."

Today the composer and the sidebar can only read a peer's state by polling _acpmux/peers. This PR adds a push. It follows manaflow-ai/cmux#26756 (merged, 85748f371e5e), which made the state honest: connected only after the handshake.

Contract

Notification _acpmux/peer_state_changed, sent to every watching connection (_acpmux/watch):

{peer: string, state: "connecting"|"connected"|"degraded"|"reconnecting"|"offline",
 lastSeen: number|null (ms), error: string|null, at: number (ms)}

When it is sent:

  • Only on a change of PeerState (peer_link.rs). It never repeats while the state stays the same, so a peer whose retries keep failing sends one reconnecting, not one per attempt.
  • It is sent from the code that changes the state. Every Link change goes through Peer::with_link, which sends the note under the link's lock, so notes go out in order. No timer, no polling.
  • A peer that is added (peer_add, config, lease, reconnect) sends its first state.
  • A peer that is removed (peer_remove, an invalid saved ssh URL, a lease that ends) sends a last offline with error: "removed".
  • A stopped instance sends nothing more. The instance that replaces it speaks for the name.

What each field means:

  • state is the same string as state in _acpmux/peers. connected means the handshake finished (#26756). degraded means the handshake is running or a ping went unanswered.
  • error is why the link is down. It is null while connected.

How a client finds it:

  • Feature peerStateEvents is in the daemon's initialize feature list, next to harnessUpdatePlan.
  • Capability peer-state-events-v1 is in initialize/_acpmux/status/acpmux capabilities.

Registration:

  • Typed in protocol/peer_state.rs (PeerStateChanged, PeerLinkState) and registered in protocol/mod.rs.
  • The IR and the generated web types are regenerated.
  • Documented in docs/acpmux-schema.json.

Ordering across instances: src/peer_states.rs keeps, per name, the instance that speaks for it and its last note, under one lock. Because of that:

  • a replaced or removed instance never speaks after its successor;
  • a reconnect that is still reconnecting sends no repeat;
  • a removal's offline/removed goes out only while the removed instance still speaks.

The notes have their own 256-slot feed, kept apart from the harness feed so a burst of notes never pushes a harness reload out of a slow watcher's buffer. A watcher that falls behind that feed (RecvError::Lagged) gets every name's last note again, so a client that sees the same state twice may ignore the second one. The table keeps one entry per running peer and the last note of at most 64 removed names; the oldest removal is dropped first.

Fan-out uses the same delivery path as _acpmux/harnesses_changed: server/fanout.rs reads the feed and passes each note to server/harness_admin.rs watch_note, which sends a payload marked peerStateChanged as _acpmux/peer_state_changed to connections that watch.

Test: fails before, passes after

tests/peer_state_events.rs a_watcher_hears_each_peer_state_change_once sets up two hubs, with A's link to B going through a TCP relay that the test drops. A watching connection on A must hear:

  1. connecting, degraded (the handshake), then connected, each once, with lastSeen/at set and error null;
  2. after the relay drops, reconnecting once with an error, and nothing more over 6 s while at least one more retry fails;
  3. after remove_peer, offline with error: "removed", and then nothing;
  4. peerStateEvents in initialize's features.

Before (#26756's head, 6fb3ca53 in that run, plus only this test), cmux-ci run https://github.com/manaflow-ai/cmux/actions/runs/40655109455:

test a_watcher_hears_each_peer_state_change_once ... FAILED
assertion `left == right` failed: []
  left: []
 right: ["connecting", "degraded", "connected"]

After (this branch, 969ac56c in that run; the next commit is rustfmt only), cmux-ci run https://github.com/manaflow-ai/cmux/actions/runs/40666930100:

peer_state_events: test result: ok. 1 passed (run 3 times, 3 passes)
acpmux --lib:      test result: ok. 1623 passed; 0 failed   (protocol_ir_is_current included)
peer_integration:  test result: ok. 50 passed
harnesses_changed: test result: ok. 1 passed
cargo clippy -p acpmux --all-targets -- -D warnings: clean

At the head (907be1dc, after the review fixes): cmux-ci run https://github.com/manaflow-ai/cmux/actions/runs/40671765409 passed fmt, peer_state_events 3 out of 3, peer_integration 50 out of 50 and clippy. In that run, lib had one failure in an unrelated test, cli::codex_auth_broker. Run https://github.com/manaflow-ai/cmux/actions/runs/40672480655 then passed lib 1624 out of 1624, including the new unit test removed_names_are_kept_up_to_a_bound.

Review

An independent Codex review (gpt-6-astra, high effort, fast mode) ran in four rounds, and each blocking finding was fixed:

  • lease-end ordering;
  • a stopped instance emitting late;
  • a repeated reconnect note;
  • lost notes after watcher lag;
  • the harness feed shared with the notes;
  • unbounded tombstones.

The last round, at 907be1dc, found nothing and returned VERDICT: SHIP.

Capture

No capture: no UI changes. The only webview files are the regenerated protocol types (generated/methods.ts, generated/types.ts), which add a type for the new notification. Nothing renders it yet; the composer lane consumes it.

Overlap

pr-overlap names #26747, #26700 and #24797, all from the same account. They share only the registry, spec, capability and feature-list lines, plus Peer's field list. The second one to land rebases and re-blesses the IR. None of them adds a peer-state push.

Risk

Medium-low. This adds one notification to watchers and changes no existing message.

  • with_link now also locks the PeerStates table, in the order link, then table (with Peer::states/last_error read inside). Nothing takes them in the other order, and the table never takes another hub lock.
  • The handshake now sets ready and connected together under the link lock, and only while the link's epoch is still current. A link that dropped mid-handshake is therefore never marked ready (this tightens #26756).
  • last_error is set before the link goes down, so the reconnecting note names the reason.

Agent session: agent session

šŸ¤– Generated with Claude Code