PR #28503 - manaflow-ai/cmux

prompt_supersede::a_prompt_queued_before_a_restart_is_still_replaced failed about once in two full hub_integration runs, at prompt_supersede.rs:257 ("the cut turn did not go on"). This PR finds the race in #27921's restart continuation and fixes it, along with the handoff gaps that independent review found around it.

The race

At a restart, the shutdown hands a running turn to the next daemon, which continues it (restart_resume.rs): the next daemon's log load finds the turn open, with no turn_result, and queues {prompt}~resume-1. The old daemon's own turn task still ran once its agent ended. finish_turn wrote a turn_result failed after the shutdown had returned. In the failing dump this was seq 19 status idle, then seq 21 turn_result failed. When that write landed before the next daemon read the log, the turn looked ended and no continuation was queued. In production the old daemon releases daemon.lock before its process exits, so the next daemon can read the log while the old one is still writing.

The fix

A handed-on turn writes no end of its own. The shutdown marks the turn it hands on (settled_by_shutdown). finish_turn also returns before writing an end, putting messages back or holding the queue when the agent's end came with or after a handing-off shutdown (Hub::hands_off). It does this even when it took the turn first. The dying daemon schedules no auto-resume of its own.
Nothing reaches the log after the shutdown. shutdown_all seals the store once agents are handed off and flushed (Store::seal). Every LocalStore write read-holds the seal lock, so a write lands before the seal returns or not at all. A turn whose answer arrives later stays open in the log, and the next daemon continues it.
Joined agent messages are delivered once. A cut turn of agent messages is continued with every message's words: the turn's user_message holds them all. So the old daemon no longer puts them back in its queue (digest::forget_joined), which would have run them twice. Quit Everything (endAgents) is unchanged: the turn ends cancelled and they go back, as before.
Agents kept for the next daemon stay alive.
- detach_child takes the child lock first and checks the handoff under it. Only then does it end prompts or kill the agent. This covers route switches, update restarts, restart_unproven, failover and the model switch at spawn.
- restart_on_route during a shutdown answers with the shutting-down error instead of "applied now".

Tests (each committed red before its fix)

Test	Before its fix	At 9b4bad7
`prompt_supersede::a_turn_a_restart_handed_on_records_no_end_of_its_own`	red ×3 (18b489a)	green
`hub::shutdown_tests::a_turn_whose_end_was_under_way_at_a_restart_is_continued_by_the_next_daemon`	red ×3 (b96cd50)	green
`hub::shutdown_tests::a_turn_ending_after_a_shutdown_began_leaves_the_kept_agent_to_the_next_daemon`	red (16f85ae)	green
`hub::shutdown_tests::a_turn_answered_after_the_shutdown_writes_nothing_and_is_continued`	red ×2 (149236d)	green
`hub::shutdown_tests::a_detach_after_a_handing_off_shutdown_leaves_the_kept_agent`	red ×2 (149236d)	green
`hub::shutdown_tests::a_cut_turn_of_agent_messages_puts_none_of_them_back`	red ×2 (97113b3): b's prompt never answers, put back in the old daemon's queue (timeout at line 239)	green
`prompt_supersede::a_prompt_queued_before_a_restart_is_still_replaced`, 64 parallel × 4	7–8 of 256 failed (base)	0 of 256 failed

Command: cargo test -p acpmux --lib hub::shutdown_tests and cargo test -p acpmux --test hub_integration prompt_supersede, on fleet CI (cmux-ci test).

Observed / Not run

Observed, at 9b4bad7 (run 41133136142): the full lib suite (2352) and hub_integration (621) passed twice each, with clippy clean and the panic ratchet green. The only other failure was hermetic_daemon_guard::every_test_binary_runs_in_the_hermetic_environment, which also fails on base.
Observed, at 5471413: the full lib and hub_integration suites passed twice, with clippy clean. Two tests this PR doesn't touch each failed once: daemon::state_file_tests::managed_unit_start_matches_shortened_sockets and agent_host::orphan_exit::a_daemon_connecting_while_the_harness_ends_is_told_it_exited.
Independent review: gpt-6-sol reviewed six rounds and gave SHIP at 9b4bad73512dc77d13ae6ca5e1c457a927dd1d0e.
Not run: a real daemon restart on a Mac with hosted agents; this covers the in-process and unhosted paths. Hosted agents keep running and are adopted, as before.
Left for a follow-up, since base has it too: on Quit Everything, finish_turn can take the turn before settle_cancelled sees it, so that turn ends as a normal failure instead of a cancel. The timeout fallback in daemon.rs (a flush after SHUTDOWN_BUDGET) is not sealed.

Changelog

Fixed: a turn a daemon restart cut now goes on in the next daemon. Before, it ended as failed whenever the old daemon's write of its end landed first.

Agent session: agent session

🤖 Generated with Claude Code