fix(broker): bound and retry the Relaycast startup handshake

Merged
WI
4mo ago
+446-13

Summary

Fixes the flaky failure in the publish workflow's macOS standalone lifecycle smoke (verify-standalone-macos), which intermittently failed with:

Broker process exited with code null during initial handshake → Broker startup readiness line missing or duplicated: expected 1, got 0

Root cause (a hang, not a crash): connect_relay performs the mandatory Relaycast handshake via the relaycast SDK's workspace bootstrap (create_workspace / agent registration). Those bootstrap calls build a timeout-less reqwest client, so when the connection to the relay backend stalls, startup_session_set_with_options hangs indefinitely. The broker never prints Broker started., and because the CLI's node up is still within its 45s handshake budget, the standalone smoke's own sleep 8 && down --force teardown then kills the still-starting broker — surfacing to the SDK as an opaque code null (signal) exit "during initial handshake".

code null (killed by signal) rather than a clean error exit (1) or a Rust panic (101) is what pointed at an external reap of a hung process rather than a code fault. This was confirmed flaky: the same run/commit was re-run and passed.

Fix: Wrap startup_session_set_with_options in a per-attempt tokio::time::timeout with bounded retry + exponential backoff. A stalled attempt now aborts fast and retries in-process, well within the SDK's 45s startup budget, instead of hanging until something reaps the process. Terminal failures/timeouts surface as clean, contextual errors instead of a silent hang.

  • Defaults: 5s per attempt, 4 attempts, 250ms → 2s backoff (worst case ~22s < 45s SDK budget).
  • Overridable via AGENT_RELAY_HANDSHAKE_TIMEOUT_MS and AGENT_RELAY_HANDSHAKE_ATTEMPTS.
  • Startup-debug tracing logs each attempt/timeout/retry for future diagnosis.

Files: crates/broker/src/runtime/session.rs (+ CHANGELOG.md).

Follow-ups (not in this PR)

  • The upstream defect lives in relaycast sdk-rust: create_workspace/lookup_workspace should build a client with an explicit timeout (the authenticated client already uses 30s). Fixing that needs a crate release + version bump.
  • The "standalone" smoke isn't truly standalone — it depends on production Relaycast reachability and mints a throwaway workspace each run. Pointing it at a local RELAY_BASE_URL stub would remove the network dependency entirely.

Test Plan

  • Tests added/updated — unit tests for the AGENT_RELAY_HANDSHAKE_TIMEOUT_MS / AGENT_RELAY_HANDSHAKE_ATTEMPTS env overrides (default/invalid/zero/valid)
  • Manual testing completed — cargo check, cargo clippy (clean), cargo fmt, and cargo test -p agent-relay-broker --lib handshake (2 passed) on the broker crate

Screenshots

🤖 Generated with Claude Code

https://claude.ai/code/session_01WzkvFAUELJgDQgZ3pdoRTP

Generated by Claude Code