acpmux: a failed log load no longer hangs attach

Merged
TE
17d ago
+59-3

Follow-up to #22916 review.

Root cause

Since #22916, _acpmux/attach, _acpmux/info, queue_edit and queue_remove (server/requests.rs), like prompts before them, wait for the startup chat-log load (Hub::wait_logs on logs_ready). load_logs sets logs_ready as its last step, and load_logs_async throws away the JoinError of the blocking task. A load that panics never sets it, so every one of those calls waits forever and the panic is never logged.

Fix

load_logs holds a drop guard (LogsReady) that sets logs_ready however the load ends, so the waiters wake on a panic too, and logs the panic as an error in the daemon log. load_logs_async now logs a failed blocking task (JoinError) and opens the gate as well, for a task that never ran its body.

State after a failed load: the daemon keeps running and answers from the session list it read before its ready line, plus whatever the load finished before the panic (some log tails caught up, maybe some queues restored). That is the state a daemon with no chat logs would answer from. Queues, cut-off turns or tails the load did not reach are missing for this run; the logs on disk are not touched, so the next start reads them again. A daemon that hangs every attach and prompt is worse than one that answers with less. Stopping the daemon instead would turn one bad log into a crash loop at every start.

Test

attach_and_prompts_survive_a_failed_log_load (tests/durable_agents.rs): a daemon restarts with ACPMUX_TEST_PANIC_IN_LOAD_LOGS (debug builds only) making load_logs panic; _acpmux/attach and a prompt must answer within 30 s.

Red, on the test commit 484568cf37d (base 3b1e3bb51b5 plus the test and its seam):

cmux-ci run --class isolated --script scripts/ci/cmux-tui-rust-check.sh --ref 484568cf37d --arg=test --arg=a_failed_log_load (step affb617238b780fe83b0aad5): FAIL after 30.99 s, failing alone too. The load panicked at hub/mod.rs:551 and then daemon answered in time: Elapsed(()) at durable_agents.rs:261; attach never answered.

Green, on head da645414b19: the same command (step c3e998826c2d91062eda8ed8) passes in 0.51 s.

Observed / Not run

Observed (fleet, aws-m4pro-2, head da645414b19):

  • a_failed_log_load: red on 484568cf37d, green on head (above).
  • survive (fd8490a004d885ed8c2c7cfd): 12 crates, 0 red (acpmux 14/14).
  • restart (c5566e52eb88d09f60253661): 7 crates, 0 red (acpmux 36/36). Two cmux-tui terminal_host_recovery tests were flaky (passed on retry), not acpmux.
  • attach (13da342c4a85f89fb30e68bf): acpmux 11/11 pass. Red: cmux-tui-core attachment_tests::bad_types_sizes_and_hashes_are_refused_before_any_bytes and three cmux-tui terminal_host_recovery host_scrollback::sized_attach tests. The same four fail the same way on base 3b1e3bb51b5 (step 3ca15ad3262001f55fbf1c6a), so they are base reds, not this PR's.
  • scripts/cmux-next/prepush-gate.py: no new violations on both pushes.

Not run: the full acpmux suite, and a real daemon with a corrupt log on a Mac. The panic is made by the test seam, not by a real bad log. A JoinError from a blocking task that never ran (runtime shutting down) is handled by reading the code only.

Changelog

  • Fixed: the agent daemon no longer leaves chats hanging when reading the chat logs at startup fails.

Agent session: cosmic-gray

🤖 Generated with Claude Code