fix(desktop): auto-recover unresponsive host-service with a health watchdog

Closed
SA
1mo ago
+681-98

What & why

Related to #5699 — "Failed to run preset / Failed to fetch (127.0.0.1:48135)". (The primary root cause — a node-pty fd leak that exhausts the system-wide pty cap — is fixed in #5714; this PR fixes the secondary failure mode where a wedged host-service is never detected.)

The HostServiceCoordinator only listens for the child process exit event. A host-service that wedges without dying (blocked event loop, fd exhaustion after extended usage) therefore stays "running" forever: getConnection keeps handing the renderer the same port, every terminal.createSession fetch fails with Failed to fetch (127.0.0.1:<port>), and the only recovery is the manual tray restart — or rebooting the machine, as the reporter did. (The port in the report, 48135, is in the coordinator's stable-port range 48000–48999, confirming the renderer was talking to a local host-service the coordinator still considered healthy.)

This adds a health watchdog to the coordinator:

  • Probes every "running" instance's /trpc/health.check every 15s (3s probe timeout), skipping instances with a pending start and discarding probe results if the instance was stopped/replaced mid-probe.
  • After 3 consecutive failed checks, force-restarts that org's host-service via the existing reset() recovery path, escalating to SIGKILL on the wedged pid (a wedged process can ignore stop()'s SIGTERM).
  • Terminals survive the restart through the pty-daemon adoption path createSession already implements; renderer picks up the fresh connection through the existing 5s getConnection poll + status events.

Related: #5700 retries transient network blips on the create path; this PR addresses the persistent-until-reboot failure mode by making the coordinator detect and heal a wedged child.

How I tested it

  • Added 6 unit tests in host-service-coordinator.test.ts covering: healthy probe clears the failure counter; no restart below threshold; force-restart at threshold with SIGTERM→SIGKILL escalation; skip when no spawn config (logged-out); non-running instances not probed; probe failures don't count against a replaced child.
  • bun test apps/desktop/src/main/lib/host-service-coordinator.test.ts — 12 pass (6 existing + 6 new).
  • bun run lint and tsc --noEmit (apps/desktop) pass.

Checklist

  • PR title follows conventional commits (type(scope): subject)
  • bun run lint and bun run typecheck pass (CI fails on lint warnings too)
  • "Allow edits from maintainers" is checked on fork PRs
Open in Stage

Summary by cubic

Adds a health watchdog to the desktop HostServiceCoordinator to auto-recover unresponsive host-service processes. It detects wedged, still-running children and restarts them, removing persistent “Failed to fetch (127.0.0.1:PORT)” errors without manual restarts.

  • Bug Fixes
    • Probe running, owned instances’ /trpc/health.check every 15s (3s timeout); clamp fetch/sleep to the remaining budget; avoid overlapping ticks.
    • Bind failure counts to the child pid; ignore stale probe results; clear on success or deliberate stop().
    • After 3 consecutive failures, force-restart via reset() and SIGKILL the wedged pid before respawn; skip adopted (non-owned) instances.
    • Revalidate after config awaits; keep counters armed when config is missing; retry respawn when recovery fails or no instance is tracked without overriding a concurrent stop()/start.
    • Remove ghost “starting” entries on spawn errors; watchdog tick logs and swallows its own failures; wire enableHealthWatchdog in main with a shared auth/config provider; tests cover thresholds, pid-binding, retries, escalation ordering, revalidation, adopted-skip, and startup failure cleanup.

Summary by CodeRabbit

  • Reliability Improvements
    • Added a health watchdog that periodically probes running host services and recovers stalled instances by force-resetting them after consecutive failures.
    • Recovery includes retries when no healthy instance is available, and escalates from graceful termination to force-kill; intentional stops now prevent auto-respawn.
    • Improved health-check timing to ensure probes and retries always respect the overall timeout budget.
  • Development Experience
    • Streamlined development reload by reusing shared auth/API configuration for host-service startup.
  • Tests
    • Expanded watchdog coverage for thresholds, replacement edge cases, retry behavior, and cleanup/disarm scenarios.