acpmux: Stop really stops an agent that works through the cancel

Merged
TE
6d ago
+721-69

Leo (2026-10-29): "are these messages literally not getting to you ... i literally can't stop you i clicked the stop button and it didnt do anything so this is still busted didnt we say we were going to fix this in acpmux? otherwise i have to manually just click steer on all of them unless you come up with better solulus"

What happened

A coordinator chat had 319 queued prompts, all from other agents. It read state=running and also stuck (harness_unavailable, a 502 from the gateway) while it kept making tool calls. Leo pressed Stop. cmux showed "The agent didn't answer Stop, so cmux ended its turn", but the agent kept making tool calls and the chat stayed running.

Why the chat still read as running after the forced end (from the code)

Mostly the queue. The agent's events alone could not keep or reopen a turn:

  • Events cannot open a turn. No inbound path creates a TurnInfo or sets Running. on_update/note_activity attach to session.turn only if one exists, and Running is set only by run_prompt (turns.rs), adoption and permission answers.
  • A queued prompt started the next turn immediately. cancel.rs's forced end (force_cancel after CANCEL_GRACE) settles the waiting prompt. finish_turn then takes the turn and notifies queue_moved. The head queued prompt's task, already blocked in queue_turn on turn_lock, takes the lock at once: nothing in the queue checks for a Stop. That head was an agent message, so take_joined pulled in up to 128 KiB of the other agent messages, and the new turn set Running.
  • The old turn's work then landed in that new turn. The harness process was still in the stopped turn. Its tool calls attached to the new TurnInfo.doing, and its heard signs kept the new turn from ever reading silent or stalled. To the person it looked like Stop did nothing.
  • The comment on cancel_preserving_queued was wrong. It said "A plain Stop withdraws every queued prompt, including one promoted with Send now". On base, only the compacting branch withdraws anything (remove_queued_because(.., "cancel_forced")). A plain Stop withdrew nothing, so the queue drained straight on.
  • Stuck could not clear while the agent worked. HarnessFailures resets only on a completed turn, and the agent never completed one.

The fix

  1. Escalation for an agent that works through the cancel (hub/cancel.rs escalate_stop). If the model kept sending signs of work through the 10 s grace (Session::heard moved since the cancel), acpmux sends session/cancel again. That is the adapter's own interrupt:

    • claude-stdio writes Claude Code's control_request {subtype: interrupt};
    • the Codex adapter sends turn/interrupt for the thread and its subagents;
    • other ACP adapters get session/cancel.

    No adapter has anything stronger short of ending the process. If the agent still works 5 s later (INTERRUPT_GRACE), acpmux takes the agent out of the slot, marks it end_on_purpose (no failover, no continuation) and calls ChildAgent::kill. That kills only the session's own recorded process group (SIGTERM, then SIGKILL, plus its setsid escapees). For a hosted agent it sends the host's Terminate. The turn ends as cancelled with cancelForced and restarted: true.

    The agent then starts again on the same conversation through the existing child_for path (--resume / session/load). This happens at the end of run_prompt, while the stopped turn's lock is still held, so the next queued turn finds it running. An agent that goes quiet keeps today's plain forced end.

  2. Never ended while it still works. From the first Stop until the turn really ends, the row's state reads stopping (a stopping record, which watchers hear as status). Updates that the ended process still had in flight (by its agent generation) are dropped in the inbound loop, so they cannot reach the next turn.

  3. Queued agent messages after the person's Stop. A session/cancel from the person's connection (Hub::person_stop) holds the other agents' queued messages (fromAgent / agent provenance). They stay queued, marked held, and are logged as queue_paused with reason: "stop". Agent messages that arrive while the hold lasts are held too. The hold ends when:

    • the person's next turn completes (release_held_agent_messages);
    • the queue is resumed.

    A failed turn's automatic retry leaves them held while the hold lasts, and an inbox digest (acpmux session inbox --digest) of them waits held and does not steer.

    Nothing is dropped, and the person's own queued prompts run next as before. The compacting branch now withdraws only the person's prompts and holds the agents' messages instead of dropping them. Stops from agents (acpmux send --interrupt, sub-session stop, remote floor) keep today's behaviour, and the misleading comment is corrected.

  4. Stuck clears on work. A session marked stuck is unstuck on the first live tool_call, thought or plan update from its agent (not a session/load replay). Its failures are forgotten and an unstuck record is logged, which watchers hear as status.

Tests (red at the test-only commit, green at the fix)

  • stop::an_agent_working_through_stop_is_restarted_on_the_same_conversation: the fake agent ignores every cancel and emits a tool call every 100 ms. The test checks:
    • the row reads stopping;
    • the steps are interrupt then restart, then the turn ends cancelled/cancelForced;
    • the new process resumed the same agentSessionId;
    • no tool call from the old pid lands after the turn ended;
    • no turn runs afterwards;
    • both agent messages (one queued before the Stop, one after) stay queued and held.
  • stop::after_the_persons_stop_their_prompt_runs_next_and_agents_wait_for_it: the person's queued prompt runs next, the agent's message is held during it and goes in after it.
  • stuck_liveness::a_stuck_session_whose_agent_works_again_is_unstuck_at_its_first_tool_call.
  • Existing digest tests that stopped the busy turn only to release agent messages now stop it as an agent would (hub.cancel), since the person's Stop holds those messages.
runcommitresult
red7789cf55fa9 (tests only)run 40970757260: all 3 new tests fail
green79a19a00289 (focused: agent_digest, stop, stuck_liveness, 5 loops)run 40975402953
red (retry vs Stop hold)20c35c757d5 (test only)run 41012944541: stop::after_the_persons_stop_a_failed_turns_retry_leaves_agent_messages_held fails
green (retry vs Stop hold)ee6cc9384f1run 41012946423: the three stop::after_the_persons_* tests pass
clippy + full crate84c74f4b30c (final head)fleet Linux run: clippy clean, every hub_integration test passes. Only hermetic_daemon_guard::only_the_hermetic_helper_starts_the_acpmux_binary (also red on base) fails, plus two daemon::state_file_tests::managed_unit_* lib tests. Those two fail only on the fleet pool's environment, pass in every Blacksmith run of the crate, and are in code this PR does not touch (src/daemon/).
base redsbase 7789cf55fa9 / c62a8dbf5a5run 40973408643: the hermetic guard fails the same way on base

PR CI reds at a7f4d7949cd, each also red on base (re-checked at the final head before landing):

  • cmux-next checks, "Prebuilt web bundles install and stamp". tests/test_build_web_bundles_from.sh hits a FileNotFoundError on webviews/package.json: since #27680, web-bundle-key.py reads that file and the test's scratch repo has none. The same failure on base tip 284e6919907 is run 41007450680. It runs here only because this PR touches a webviews file (the generated protocol types).
  • cmux-next checks, "Package dependency lists sorted", "Script tests" and "Sidebar look owner". All are red in base cmux-next runs 41004771340 and 41005881456. The signal 5 annotation comes from the swift-test-summary.test.sh fixture, and this PR changes no Swift.
  • release-notary and Guard status. The same two failing tests as base guards run 41006843794: the glaeda signing hook list and the What's New highlights. Guard status only aggregates.

Review

Independent reviews ran with Codex gpt-6.1-sol at high, with fast mode on, through the team route. The coordinator allowed this after gpt-6-astra hung on two GCP lanes tonight. Round by round:

  • BLOCK findings fixed:
    • the stopped agent's child is now re-read before escalating;
    • the killed agent's taps are muted;
    • the cancel epoch is kept for the respawn;
    • held agent messages now include ones sent now;
    • the idle-race hold;
    • the hold now has a number per turn, so a finishing turn cannot release a newer Stop's hold;
    • the hold is checked and ended under the queue lock;
    • after the base merge of #27536 (carried turns), escalation follows a carried turn, and a carried turn's Stop restart starts the agent again.
  • After the base merge of #27523 (inbox digest):
    • a digest made while a person's Stop holds agent messages now waits held and does not steer;
    • a failed turn's automatic retry no longer ends a person's Stop hold. Only the person's next completed turn or a resume ends it.
    • #27523's tests that ended their busy turn with the person's Stop now end it with an agent's Stop, as this PR's other digest tests do.
  • Final verdict: SHIP at 84c74f4b30c.

Risk

  • Restart cost. An agent that keeps producing model output for 15 s after Stop is restarted. A long tool call that the harness does not interrupt but that also sends no model signs is not restarted (heard counts model signs only, not tool heartbeats beyond Claude's tool_progress).
  • Hold length. After the person's Stop, agent messages (and an inbox digest of them) wait for the person's next completed turn or a queue resume. A daemon restart forgets the hold.
  • New state value. state: "stopping" is a new value of an existing string field.
  • Size. About 790 changed lines, tests included.

Overlaps (same author, coordinated by the acpmux area owner): #27536 changes cancel.rs force_cancel / cancel_preserving_queued (carried turns), and #27038 / #26747 change queue.rs enqueue.

Agent session: agent session
šŸ¤– Generated with Claude Code