#22916 acpmux: queued prompts and a dequeued prompt survive a daemon crash

Summary

Two acpmux crash-recovery bugs in the prompt queue.

Bug 1: an empty queue after a restart. Since fa66d79687f the daemon answers right after its ready line and reads the chat logs afterwards (Hub::load_logs: log tails, the turns a restart cut off, the queues). _acpmux/attach and _acpmux/info did not wait for that, so they answered from the session list alone. A chat attached right after a crash showed queue: [], status idle, turn: null and a stale lastSeq. A queue_edit or queue_remove in that window found no prompt to change. The failing reply shows the race: the session detail says lastSeq 10 while the same reply's top-level lastSeq (read a moment later, after catch_up) says 19. Fix: attach, info, queue_edit and queue_remove wait for wait_logs(), as prompts already do (server/requests.rs). The session list still answers at once.

Bug 2: a dequeued prompt was lost. restore_queue dropped a queued prompt at its dequeued record. When the daemon died after the dequeue and before turn_started, the prompt had no queue entry and no result. A resend (_meta.acpmux.resend) answered it from the log (logged_prompt) as interrupted, so it never ran and nobody was told. The prompt never reached the agent, because turn_started is logged before the prompt is sent. The fix puts it back at the head of the queue. restore_queue feeds each record it scans to a small tracker (hub/lost_start.rs). It keeps prompts that were dequeued for their own turn (not into a running turn as a steer) and have no later turn_started, turn_result, prompt_refused, queue_removed, a steer user_message or a requeue. Those go ahead of the waiting prompts, with the same ledger entry a resend waits on, so the prompt runs once. logged_prompt and the prompt_with ledger are unchanged. A dequeued prompt refused before its turn now logs prompt_refused, so a later restart does not bring it back. This covers run_prompt's check_cut and start_turn, which logged nothing before (check_dispatch already logged it). QUEUE_KINDS adds turn_result and prompt_refused.

The tracker carries a prompt's whole queued record, after any queue_updated, so fields added to it later (such as images) ride along.

Testing

Red on base 1d78eb625bd7, from scripts/ci/cmux-tui-rust-check.sh test <filter> fleet steps:

• queue_ops_edits_survive: step 41e4934b607e8d0dab9023e1 fails with left: [], right: ["q-2", "q-1"]. The same failure appeared earlier at 988e0d1 in step 1af4c9957855ecbda7d7fae1.
• queued_prompts_survive: step 7c69c0966cb590284cff5c2a. queued_prompts_survive_a_daemon_crash and queued_prompts_survive_an_update_restart fail with left: [], right: ["q-1", "q-2"].
• a_dequeued_prompt_survives (the new test) at the red commit 685c042b352: step 81528b5f2d6bc91c5a64fe15. The resend gets {"stopReason": null, "_meta": {"acpmux": {"status": "interrupted", "duplicate": true}}}.

Green on the fixes:

• At 6626f79 (bug 1 only), filter survive, step b1e78f8f1a04b4b6551ce1ba: queue_ops_edits_survive_a_daemon_crash and all four queued_prompts_survive_* pass. Only the new bug 2 test is still red.

• At 2b55165 (both fixes), filter survive, step f96274da76476c0d675c190d: 114 of 114 pass, including all 13 acpmux tests (a_dequeued_prompt_survives_a_daemon_crash, queue_ops_edits_survive_a_daemon_crash, queued_prompts_survive_*).

• At the head 7e1fd2d, all acpmux tests pass under three filters:

  • restart: 33 of 33, step 2fc9d8915abcc1dcce1f298b.
  • adopt: 30 of 30, step 95907660b5a594abbea4c951.
  • attach: 10 of 10, step 0696b84f560a9f89c6d18156.

  The cmux-tui reds in those runs are terminal-host and credential tests; the restart ones passed on retry.

• At e7f7c83, which is this code plus the held send-now commit described below, all acpmux tests pass under three more filters:

  • survive: 13 of 13, step e6c7422d6a1e8e1cef0bc177.
  • queue: 31 of 31, step 894b808a0b214fa417deeedb.
  • steer: 26 of 26, step 072e2459411e22fcb843d975.

Observed / Not run

Observed:

• The red and green fleet steps listed above.
• A full all run (4 shards) at 2b55165. Its acpmux failures are in areas this PR does not touch: claude_auth_env, no_keychain, web/start modes, session_pool hello, codex processes, subsessions, session_offers, and the hermetic guard on startup_load/workspace_roots. Two failures got a closer look:
  • transfer_guard::a_chat_with_no_asking_mode_moves_under_peer_control_but_not_under_web fails the same way on base: steps 27fd2e0f181ac538980f254a (base) and 1d89c970663ecfa892758272 (head).
  • startup_load timed out only under that loaded shard. It passes alone at base and head: steps 5a086726f2bd0fe85d413323 and 24820608694860f6ac923078.
• That run's clippy errors are in steer_now_tests.rs and agent_roster_fold.rs, which this PR does not touch.
• The cmux-tui agent_message_pump/policy reds are credential_owner_only refusals in cmux-tui-core, not acpmux.
• CI crash ratchet: the acpmux exit hit is the debug-only test seam, now a reviewed crash-allow. The CmuxNextDesign dynamic_dispatch hit and the defaults-snapshot failure come from base; this PR touches no Swift.

Not run, or only read:

• The other acpmux failures in the full run were not each re-run at base.
• No live app dogfood.
• Attach now waits for load_logs right after a restart. On a very large store (the 3.9 GB case behind the deferral) a chat opened in that window waits for the logs. The sidebar list does not wait.
• Held back for coordination with the queue lane: queue_slot's send-now steer arm should also log prompt_refused when steer_queued fails before reaching the agent (commit e7f7c83, not on this branch). Until then, if a send-now steer is refused in the window where the log shows no open turn, a later restart can queue it again.
• If the daemon dies between user_message and turn_started, the rerun records a second user_message with the same turn id, so history can show that prompt twice.
• A never-queued prompt that dies in that window still answers interrupted on resend. Its full prompt is not in the log.

Changelog

• Fixed: prompts queued in an agent chat, and a prompt that was about to start, are no longer lost when the acpmux daemon crashes or restarts.

Agent session: cosmic-gray

🤖 Generated with Claude Code

Leo-ask: ask-33297758a3
Leo-ask: ask-bc141f48b9