fix(cli): harden fleet-node attach against transient reachability

Merged
KH
2mo ago
+661-30

Summary

  • retry the terminal-session POST only for structured, safe-to-repeat node_unreachable and terminal_session_unavailable responses (three bounded attempts total)
  • do not retry timeouts, network failures, node-not-found responses, malformed responses, or unclassified failures where the non-idempotent POST may already have completed
  • add node reference/resolved ID, redacted endpoint, timeout, HTTP/code, attempt count, and retry rationale to attach errors
  • extend established-session resume coverage from 15.5s to 31.5s so it spans the node terminal lane's independent 30s reconnect ceiling
  • bound local WebSocket close diagnostics to RFC 6455's 123-byte limit so reconnect exhaustion closes instead of hanging
  • keep the existing SSH fallback unchanged

Fixes #1571.

Root cause

The initial error and the later 1011 close are part of the same transient terminal-access incident class, but they have different immediate origins:

  1. The exact Node '<name>' is not reachable text comes from Cloud's terminal-session POST route. Its D1 node lookup succeeded, then the 45-second heartbeat liveness gate returned structured node_unreachable/503 before session allocation.
  2. The established-session terminal transport could not reconnect to the fleet node text is emitted by the CLI loopback proxy after its remote terminal WebSocket resume budget is exhausted. The old 15.5-second client budget could expire before the broker's independent terminal lane, whose backoff reaches 30 seconds, recovered.

One premise in the investigation brief was therefore not quite correct: the 1011 is server-side only from the local attach client's perspective. The server emitting that close is the CLI's loopback proxy, not Cloud or the fleet node; a remote terminal-socket close triggers it after local resume attempts are exhausted.

Reproduction evidence

Before the fix, using the installed 11.6.10 CLI against finn-mini, I ran 20 unchanged --mode view attaches with 4-second spacing and a 6-second observation window. 18 stayed attached; attempts 18 and 20 failed with Node 'finn-mini' is not reachable. Immediately after attempt 20, the node query reported the same node online/live with six agents and a fresh heartbeat.

With this branch's built CLI, the same target completed 20/20 attach observations with 2-second spacing and 10-second windows. This sample shows the observed flake disappeared; it does not claim that every successful run exercised a retry.

Verification

  • npx vitest run packages/cli/src/cli/lib/attach-fleet-node.test.ts packages/cli/src/cli/commands/fleet-agent.test.ts — 38/38 passed
  • must-fire proof: the transient-503 recovery test fails when retries are temporarily set to zero and passes when restored
  • must-not-fire coverage: permanent node_unreachable stops after exactly three attempts; node_not_found, malformed responses, and ambiguous network failures stop without unsafe extra attempts
  • distinct sixth-reconnect recovery and exhausted-reconnect close tests pass; the test server now counts distinct sockets instead of returning a consumed connection twice
  • npm run typecheck and final CLI tsc --noEmit — passed
  • npm run build:cli — passed
  • npm run lint — passed (0 errors; repository warnings remain)
  • Prettier and git diff --check — passed
  • staged secret-pattern scan — clean
  • npm test — 2,069 passed, 25 skipped, one failure: broker-lifecycle.test.ts expects the legacy Reflex warning text but receives structured JSON. The identical single failure reproduces in a clean origin/main worktree, so it is not introduced here.