feat(workspace-client,desktop): relay WS outage telemetry + version-stamped desktop events

Merged
KI
3mo ago
+359-0

Summary

  • Add outage-episode telemetry to createRelaySocket (the eventBus / workspace "disconnected" surface, previously silent): relay_ws_access_denied, relay_ws_degraded, and relay_ws_recovered PostHog events, at most one per outage plus one on recovery.
  • Register app_version as a PostHog super property in the desktop renderer so every event (including the existing terminal_connect_failed) is attributable to the build that emitted it.

Why / Context

Investigating the remote-workspace disconnect reports (relay 401 redial loop, fixed by #5628/#5637 but not yet released) hit two observability gaps:

  1. terminal_connect_failed events carry no app version — once 1.14.4 ships there's no way to segment pre/post-fix builds and confirm the 401 curve dies. The person-profile desktop_version reflects the current install, not the build at event time.
  2. The eventBus relay socket that drives workspace "disconnected" state emits no telemetry at all, so access-denial episodes (the v2Hosts/v2UsersHosts class of failure) and outage durations are invisible in production.

How It Works

  • outageReporter.ts (co-located in relaySocket/) is a pure state machine: callers feed it dial outcomes (attempt, accessDenied, failed, opened) and it collapses connection churn into episodes — one access_denied (preflight 403) or degraded (≥5 consecutive failed dials) event per episode, one recovered with outage_ms and attempt count when the socket reconnects. partysocket's retryCount only resets after 5s of stable uptime, so flapping accumulates into one episode instead of spamming pairs.
  • The sink is process-wide (setRelaySocketTelemetry), keeping @superset/workspace-client analytics-vendor-free; the desktop renderer wires it to PostHog in initPostHog. Web consumers are unaffected (sink defaults to null).
  • Sockets carry a name label (event-bus, web-terminal) so consumers are distinguishable in queries. The reported endpoint is the attempt URL minus its query string — the token never leaves the module.

Manual QA Checklist

  • Desktop dev app: events carry app_version (CDP-verified: super property registered, 1.14.1 in dev)
  • Outage cycle CDP-verified end-to-end: dead endpoint → one relay_ws_degraded after 5 failed dials → healed endpoint → one relay_ws_recovered with sane outage_ms; access-denied and terminal give-up paths verified too (see verification comment)

Testing

  • bun run typecheck (workspace-client, desktop, web)
  • bun run lint (exits 0)
  • bun test in packages/workspace-client — 18 pass: new pure unit tests for the reporter policy (threshold boundary, once-per-episode dedup, silent short blips, episode reset on regrant, token stripping) plus WS integration tests proving the socket wiring end-to-end against a scriptable local relay
  • Mutation-verified: removing the degraded emission and breaking the access-denied dedup each made exactly the expected test fail

Design Decisions

  • Module-level sink instead of threading a callback through RelaySocketOptions: every getEventBus call site would need plumbing for no gain; a process-wide analytics hook is the same shape posthog itself uses.
  • Once-per-episode emission instead of per-attempt: the eventBus 403 path re-probes every 5 minutes indefinitely — per-attempt events would flood; episode + recovery events preserve the debugging signal (cause, duration, attempt count) at bounded volume.

Known Limitations

  • terminal_connect_failed in the desktop terminal transport keeps its own capture path; it now gains app_version via the super property but doesn't use the outage reporter (it has its own max-attempts/give-up semantics). Unifying them is possible follow-up.
Open in Stage

Summary by cubic

Adds relay WebSocket outage telemetry and version-stamped desktop analytics to improve visibility into remote workspace disconnects. Emits relay_ws_* events once per outage and records app_version on every desktop event; fixes attempt counting and filters synthetic closes for accurate stats.

  • New Features
    • @superset/workspace-client: added outage reporter and process-wide telemetry sink (setRelaySocketTelemetry, RelaySocketTelemetryEvent). createRelaySocket now reports access_denied, degraded (≥5 failed dials), and recovered with outage_ms and attempt count; tokens are stripped from endpoints. Hardened reporting: correct attempt ordinals, ignore partysocket synthetic closes, include last close info when errors cross the threshold, and guard the sink with try/catch.
    • Socket labeling: new name option on createRelaySocket; event bus uses event-bus, web terminal uses web-terminal.
    • Desktop: PostHog sink forwards relay_ws_${kind} events with endpoint, region, close info, and attempt counts. Registers app_version as a super property so all desktop events are attributed to the emitting build.

Summary by CodeRabbit

  • New Features
    • Added relay connection health telemetry for outages, access-denied episodes, and recovery, including endpoint context and outage duration.
    • Desktop analytics now forward relay socket outage/recovery signals to PostHog and include the application version.
    • Web terminal and event-bus relay connections now include identifiable connection labels for clearer attribution.
  • Bug Fixes
    • Improved reliability of relay outage telemetry emission, ensuring telemetry failures don’t affect socket behavior.
  • Tests
    • Expanded telemetry and outage episode coverage, including threshold crossing, recovery, access-denied semantics, and data privacy (no token leakage).