#36739 cmux-host: the relay's backoff wait no longer panics its thread (hq#7086)

Part of manaflow-ai/cmuxterm-hq#7086. Leo: "it should take <30 seconds from starting a thing to having a response on cloud".

The relay socket from #35785 never takes a chat on a new machine. On staging (image cloud-native-0cda689, daemon 1.0.0+0cda68961caf), I created a machine through the API, waited for it to run, and sent POST /v1/cloud/machines/{m}/chat. Every chat answered queued machine_offline, then cloud.relay.machine_offline after about 62 s. The machine's cmux-host.service journal shows why:

18:06:25 cmux-host: cloud: bind: bound machine=vm_5d44… epoch=1
18:10:53 thread 'cmux-host-relay' (10870) panicked at crates/cmux-host/src/cloud/relay.rs:413:40:

run() waits between dials with rt.block_on(or_stop(&mut stop, tokio::time::sleep(backoff))). The Sleep is built as an argument, before block_on enters the runtime, on the relay's own plain thread, which is in no runtime. So the first time a socket drops or a dial fails, the relay thread panics. The machine never dials the relay again for the rest of its bind.

The fix moves the wait into pause(), which builds the timer inside an async block that block_on polls. I also checked the other block_on calls in cmux-tui/ for the same mistake. serve(..) is an async fn, so it is lazy and fine, and the remaining two already wrap their timers in async {}.

This explains the relay staying dead after its first drop. It does not yet explain why the very first chats (seconds after bind) also found no host. Once this lands and an image carries it, I'll rerun the same staging probe to find out.

Test

cargo test -p cmux-host --lib cloud::relay::tests::the_backoff_wait_runs_from_the_relay_thread_outside_any_runtime

• Red: commit b184b98c (the test, with pause() unchanged) fails.
• Green: the head passes.

Both results are in the comment below.

Observed / Not run

• Observed: the staging probe output and journal lines above; red and green results from cmux-ci test at this PR's head.
• Not run: a staging machine on an image with this fix.

Changelog

• Fixed: a Cloud machine's chat relay no longer stops for good after its first dropped connection.

🤖 Generated with Claude Code

Agent session: agent session

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Fixes a panic that silently killed the cmux-host relay thread after its first dropped socket or failed dial, leaving Cloud machines permanently unreachable over chat (hq#7086). The backoff wait between dials built its Sleep timer on the relay's plain thread, which runs in no runtime, so it panicked; pause() now creates the timer inside an async block that block_on polls.

• Adds a regression test that proves the backoff wait runs from a thread outside any runtime.
• Audited the other block_on waits in cmux-tui; the remaining ones are safe.