acpmux: send --no-wait acks within a second for any machine; the daemon delivers it (send --status)

Merged
TE
15d ago
+1028-26

Coordinators message lanes on other machines with acpmux send --no-wait huge-blue/<lane> -. From cosmic-gray on 2026-10-21 that took 60-120 s and often hit timeout 60 (rc 124), even though the message was delivered. ls, info and last to huge-blue took 1.3-4.7 s, ssh 0.5 s, and local sends under 1 s. This PR makes --no-wait ack within a second for every target and hands the delivery to the daemon.

Root cause

send --no-wait (cli/output.rs:87 queue_prompt) returns when it hears _acpmux/prompt_accepted for its prompt, or when the session/prompt request answers. For a session on a peer:

  • The forwarded request answers late. server/peer_forward.rs:136 awaits peer.request(session/prompt), which the owning daemon answers only when the target's turn ends. For a steer, it answers when the agent's current step ends. A lane at work holds that for minutes.
  • The early acceptance never arrives. The owning daemon does send _acpmux/prompt_accepted on the peer link as soon as it takes the prompt (server/requests/composer.rs:89). But the forwarding daemon drops every peer notification except session/update, _acpmux/event and _acpmux/permission_pending (hub/peers.rs:519-599, _ => return). So the CLI never hears it, and waits for the whole turn or for timeout. The delivery itself already happened, which is why rc 124 still delivered.

It isn't an ssh hop per send (the link stays open), a retry on a lost ack, or queue_slot waiting. The prompt is queued or steered right away on the owning daemon; only the ack is lost.

Fix

  1. Pass on a peer's acceptance. The forwarding daemon now passes the peer's _acpmux/prompt_accepted to the client that sent the prompt as soon as it arrives (hub/outbox.rs PeerAccepts, plus peer_forward.rs and hub/peers.rs). On its own, this makes any client's send to a busy peer session ack within a second.
  2. Make --no-wait an outbox send. The CLI hands the message to its own daemon (_acpmux/send_queue, unix socket only).
    • The daemon checks that the target exists here or on a peer, records the delivery in outbox.json, and answers with a delivery id at once. That answer is the ack.
    • It then delivers in the background, over its own connection and as the sender's proven session, so every rule of a direct send applies.
    • It retries with backoff for about 34 minutes on a connection error, a peer that is down, or no answer within 120 s.
    • Every attempt uses the same prompt id and resend, so a target that already took the message answers with the first outcome and never runs it twice. A repeated prompt id returns the first delivery.
    • A refusal (no such session, the folder's trust, a bad prompt, a full agent pair) fails the delivery at once.
    • A paused agent pair (agent_pair.paused, held: true) leaves the delivery held. It is final here and never sent again; the target's daemon runs it when the person resumes the pair.
    • --no-wait waits up to 500 ms. If a refusal or hold comes back in that time, it prints it as its error, as a direct send did, so a looping agent still hears the pause.
    • A retry ignores an acceptance marked duplicate that has no turn yet (a run of the same id still in flight) and waits for that run's answer, so a held or refused first run is never reported as delivered.
    • Undelivered messages survive a daemon restart.
  3. Check delivery. acpmux send --status ID (_acpmux/send_status) reports queued, delivering (attempt n), delivered (the target's turn, steered or queued), held (a paused agent pair) or failed (the last error). --json gives the same fields.
  4. Local sends. --no-wait still prints steered or queued for a session that takes the message at once, now with the delivery id.

A send without --no-wait is unchanged, and --no-wait still steers by default.

The README's peers section now has a design note on this transport (enqueue, ack, async forward, retry and dedupe by prompt id, status), plus the send and send --status rows.

Risk: risk:normal. A new daemon method open to the unix socket only, with delivery through the same request path as a direct send.

Changelog

Fixed: acpmux send --no-wait to a session on another machine returns within a second instead of waiting for that session's turn, and acpmux send --status ID says whether the message was delivered.

Tests

cargo test -p acpmux --test send_async uses two in-process daemons, A peered to B, where B's session runs a turn that never ends:

  • a_busy_peer_session_acks_a_forwarded_prompt_within_a_second: A's client hears prompt_accepted (with peer: b) within 1 s.
  • send_queue_acks_at_once_and_status_reports_the_delivery: send_queue answers within 1 s and send_status reports delivered with the prompt id. The same prompt id again returns the same delivery, and B queued the message once.
  • send_queue_to_a_peer_that_is_down_acks_at_once_and_keeps_trying: acks within 1 s, then stays queued with a peer_unreachable error and retries. A session nobody has is refused at once.
  • a_paused_agent_pair_holds_the_delivery_and_says_so: one hub. A session sends three near-identical messages to another through send_queue, over its caller-token connection. The third settles held after one attempt, with the refusal as its error.

Hosted runs (./scripts/verify-cmux-tui-hosted.sh --filter <name>):

Observed / Not run

  • Red on base, observed on huge-blue: all three tests fail. The busy-peer test fails with no prompt_accepted within 1s; the other two fail with Method not found: _acpmux/send_queue.
  • Green with the fix, observed on huge-blue:
    • At the merged head 67bce1e922a40d5a890b457390add0ee5f46a7d2: send_async 4/4, hub_integration 370 (with base's prompt_supersede), peer_integration 30, cli_agent 5, and --lib 543.
    • cargo clippy -p acpmux --all-targets -D warnings and prepush-gate.py are clean.
    • The protocol IR (spec/acpmux-protocol.json) is regenerated.
    • These ran with umask 022; this host's group-writable default trips unrelated permission tests.
  • Not run:
    • No live send between machines; the fix reaches the live daemons with their next update, and I restarted none.
    • The outbox's restart path, which reopens outbox.json and delivers again, has no test of its own.
    • No Swift changes.

Follow-ups (from the review, not in this PR)

  • A delivery queued from the person's own CLI is sent without the person proof, so it does not resume a paused agent pair the way a person's direct send can. Today the acpmux CLI never proves the person (only hand/daemon.rs calls prove_person), so acpmux send is not affected.
  • outbox.json is written and synced under the outbox lock on every state change. This is blocking I/O on the async runtime; move it to spawn_blocking or a writer task.
  • A daemon restart restarts a delivery's retry delays from the beginning.
  • Right after a restart, before the peers list their sessions, a delivery to a peer session can fail with no session.
  • A peer that took the prompt but whose acceptance was lost on a dropped link: each retry gets a duplicate acceptance with no turn, which is ignored, and then waits for the turn's end. If that turn runs longer than about 50 minutes, the delivery ends failed although the prompt ran (it never runs twice). Fix: the duplicate acceptance says whether the first run was already accepted.
  • The restart path (reopening outbox.json and delivering again) has no test of its own.
  • --supersede with --no-wait: the replaced count is printed only once delivered (not by --status), may read ? or 0 when a retry gets the target's duplicate acceptance, and has no outbox test of its own.

Leo-ask: ask-1d17421b73

Agent session: huge-blue

🤖 Generated with Claude Code