acpmux: preserve retries and adopted prompt outcomes across restart (takes over #26981)

Merged
TE
6d ago
+2095-72

Part of HQ#5075 (PR order 2: durable retry and adopted prompt ledger) and HQ#4879 (finding 2: a turn a daemon restart interrupts is not always resumed; an auto-resume wait does not survive a restart).

Takes over #26981 (its owner session is gone), carried onto current feat-cmux-next with its commits and authorship, plus a fix for the review finding that blocked it.

Automatic retries survive daemon restart at their original deadline, with a stable prompt ID suppressing duplicate execution. Non-hosted turns interrupted by restart continue immediately; adopted running prompt IDs remain running on resend. See #26981 for the full design, its write-boundary test tables and its earlier red/green runs.

Leo: "I think the various situations we have with cloud and whatnot, like just having that done real nicely will be quite important."
Leo: "ACCEPT IT BREAKING AT SOME POINT BUT IT RESUMES WHATEVER IT INTERRUPTED. END OF STORY FOR THE DAEMON UPDATE."

What changed relative to #26981

  • Merge onto base (765 commits later). Conflicts resolved:
    • hub/auto_resume.rs: take_turn keeps base's carried-turn start (#27536, next starts in the same step) on the per-session retry mark (Session::auto_resume) this branch moves to; stopped_for_test reads that mark; set_retry keeps this branch's signature with base's JsonSet::set.
    • hub/hosts.rs: a recovered turn's finish keeps the carried next turn (#27536) and also publishes the adopted prompt's outcome (outcome.send_replace).
    • hub/restart_resume.rs: both the carried-steer scan (#27536) and the durable turn_stop_requested marker.
    • hub/auto_resume/restore.rs: tests call take_turn(.., None); the one new v["k"] = x write uses JsonSet::set (ebce6312).
    • #27040 (autoResumeDelaysMs sets the attempt count) and b0f77cc2 (Stop before a retry's turn starts) cover neighbouring ground but not this: neither restores a scheduled retry after a restart.
  • Review fixes. The earlier review of #26981 and four review rounds here found these; each has a failing test committed before its fix:
    • daemon/cloud_handoff.rs: a late retry decision could replace a finished turn's end. After a manual retry completed, end() set Now::Quiet; with Cloud away its done report waited in later; the superseded automatic retry's auto_resume_skipped became NoRetryFor, and finish() replaced the waiting done with failed. The reporter now keeps the newest turn whose end it reported (Track::ended, alongside the turn waiting in later), drops a NoRetryFor or RetryRestored for that turn or an older one while quiet, and drops an older turn's end in finish so it never removes a newer waiting end. Only a begin the store accepted or never answered, of the newest ended turn, reopens it. A RetryRestored renews a held turn's wait only when it names that turn. Tests: a_late_retry_decision_keeps_a_finished_turns_end, a_restored_retry_of_another_turn_keeps_the_held_turns_deadline, an_older_turns_end_never_drops_a_newer_waiting_end.
    • hub/restart_resume.rs: #26981 changed logged_prompt to answer None for a prompt that was queued but never started, which broke the restart continuation's once-only guard. A crash between the continuation's queued record and the cut turn's turn_result queued it twice. The guard now also counts a queued record with that ID (queued_logged). Test: a_crash_before_the_cut_turn_settles_queues_its_continuation_once.
    • hub/inbox.rs: on a restart, base's restored inbox review (#27686) could wake an idle chat with a turn of its own before the durable retry fired, so the retry was skipped as superseded. inbox_review now waits while continuation_pending, as digest.rs and subagent_retry.rs already do. Test: an_idle_chat_waits_for_its_cut_turns_continuation_before_its_inbox.
    • Panic ratchet: the Value index reads this PR added in production code use .get() (acpmux indexing_slicing stays at base's count).
    • Not changed: make_room evicting a quiet track's Track::ended. Eviction leaves later alone, which ended_here also checks, so an end still waiting stays protected. An end the store already has can't be replaced, because the store refuses any report on a turn that already ended (module doc, Outcome::Refused).

Observed / Not run

Latest, at aaeff484938c1f32107c8bd44e78c7b8079fb2e5, run 41073089483: red at 426076ea (the inbox test), green at head: lib 2,273 passed, durable_agents 131 passed, clippy clean. Earlier runs:

Run 41028785396 (Blacksmith 32 vCPU, cmux-ci test) at 13acc523f8aedc5b9dcd80463a9b65fe032c140f:

  • Red at d798debb (tests before the fix): a_restored_retry_of_another_turn_keeps_the_held_turns_deadline and an_older_turns_end_never_drops_a_newer_waiting_end fail.
  • Red at 3f483395: a_crash_before_the_cut_turn_settles_queues_its_continuation_once fails, left: 2, right: 1 (two queued records).
  • Earlier runs 41002430556: a_late_retry_decision_keeps_a_finished_turns_end fails at 46a071c9, 3977a341 and f215a828, each before its fix.
  • Green at head: cargo test -p acpmux --lib 2,027 passed, 0 failed; cargo test -p acpmux --test durable_agents 130 passed, 0 failed; cargo clippy -p acpmux --all-targets -D warnings clean.

Not run: no Swift is changed, so there is no Mac build or app capture. No two-daemon restart run on a live host (that is HQ#5075's later acceptance test).

Changelog

Fixed: automatic retries and running prompt outcomes recover across daemon restarts.

Agent session: agent session

šŸ¤– Generated with Claude Code