cmux-host: a machine that wakes from a pause redials its relay at once

Merged
TE
21h ago
+284-25

Leo, 2026-11-05: "I thought we had better typed errors than this. I thought we had a better solution than whatever was going on here."

A Cloud chat sent right after the machine wakes from a pause failed with cloud.relay.host_gone: the machine's relay connection closed. In production run cmuxterm-hq 42119371232, a pre-booted spare sat paused for 12.5 min, resumed, and cmux cloud start-chat failed after 28 s. The guest journal:

22:12:32 acpmux: this machine did not run for 12m 33s ... gap_ms=753239
22:12:34 cmux-host: updater: current        <- the updater ran its resume check and restarted nothing
22:12:53 cmux-host: cloud: report change+daemon failed

The guest's monotonic clock doesn't count a pause. So the relay (cloud/relay.rs), which drops a socket after 60 s of silence measured on tokio::time::Instant, kept treating its socket from before the pause as fine. The platform's MachineRelay sent the new chat down that dead socket. When the host finally saw the reset and dialed again, the platform retired the old socket as replaced and ended the in-flight chat with host_gone. The status report after the resume also reused the pooled HTTPS connection, so it hung for its full 20 s timeout ("report change+daemon failed" came 21 s after the resume).

On HostEvent::Resumed, cmux-host sent one resume report and nothing else. Now the worker's resume:

  • calls Relay::resumed(). The relay drops its socket without a close (a close could hang on a dead socket) and dials again at once, with no backoff and on a fresh HTTPS connection for the bearer mint. Chats running on the machine are not ended: their tasks and up queue carry over to the new socket, through up to 3 failed dials after the resume. Any other socket end still drops its chats, as before. A resume also cuts a backoff wait short.
  • resets the live client's pooled HTTPS connection (Http::reset, JsonPoster::reset) before the resume report, so the report doesn't hang on it.
  • as a backstop for a missed Resumed: a wall-clock jump of more than 60 s between two ping ticks (20 s apart) counts as a resume.

relay.rs changes stay inside Relay, run, pause and serve, away from final_event/run_chat (#36828 changes those). The new tests are in their own file, relay_resume_tests.rs. The GitHub minter's and trajectory mirror's own clients are not reset here; they recover through the existing rebuild-on-failure path.

Changelog

  • Fixed: a Cloud chat right after a machine wakes from a pause no longer fails with "the machine's relay connection closed"

Tests

The test commit a8ab3a13a1e adds the tests plus the hooks as no-ops, so it compiles and fails on assertions, not on a missing symbol. The fix is ad444574075.

  • cmux-host cloud::relay::resume_tests::a_resume_drops_the_socket_at_once_and_keeps_its_chats: a local WebSocket endpoint takes the socket and stays silent. After a resume, serve must return Ended::Resumed within 2 s, the endpoint must see the socket close, and a running chat must stay alive.
  • cloud::relay::resume_tests::a_resume_cuts_the_backoff_short, a_wall_clock_leap_between_pings_reads_as_a_resume
  • cmux-server cloud_http::tests::a_reset_drops_the_pooled_connection: a keep-alive server counts dials. Two GETs use one connection; after reset(), the next GET makes a second.

One cmux-ci test run (fast-checks run 42124707624) checked out each commit and ran cargo test -p cmux-host --lib -- cloud::relay:: and cargo test -p cmux-server --lib -- cloud_http::tests::a_reset:

=== RED test commit a8ab3a13a1e
test cloud::relay::resume_tests::a_wall_clock_leap_between_pings_reads_as_a_resume ... FAILED
test cloud::relay::resume_tests::a_resume_drops_the_socket_at_once_and_keeps_its_chats ... FAILED
test cloud::relay::resume_tests::a_resume_cuts_the_backoff_short ... FAILED
test result: FAILED. 16 passed; 3 failed; ... finished in 10.00s
test cloud_http::tests::a_reset_drops_the_pooled_connection ... FAILED
assertion `left == right` failed: after a reset the next request dials a fresh connection
EXIT cmux-host=101 cmux-server=101
=== GREEN head ad444574075
test result: ok. 19 passed; 0 failed; ... finished in 0.07s
test result: ok. 1 passed; 0 failed; ... finished in 0.01s
EXIT cmux-host=0 cmux-server=0

The command exited 0 (FAST_CHECKS_RECEIPT ... "exit_code": 0). The job is marked failed only because the script checked out other commits, so the head after the run differed from the snapshot. The red exits above are the real result.

Observed / Not run

Observed: the cmux-ci test run above. An independent Opus review returned SHIP at ad444574075. Not run: a live pause and resume of a Cloud machine. The role-level wiring in control_words (two calls) is covered by review only.

Agent session: agent session

šŸ¤– Generated with Claude Code