feat(desktop): re-land fleet resource telemetry as a daily digest
Summary
- Re-lands the SUPER-1550 fleet memory telemetry that #5964 removed, redesigned so it cannot repeat the ingestion problem: 5-minute samples aggregate locally and upload one
resource_digestevent per machine per UTC day (p50/p90/max per metric), plus rare cooldown-limitedresource_alertevents. - Extends coverage to the blind spots the old
resource_snapshotnever had: child-process trees (host-service, ptys, agents), host memory pressure, main-process event-loop delay, and window-unresponsive episodes. - Makes renderer
unresponsivereporting production-grade (Sentry error + digest counter). Today a frozen window in production emits no signal at all.
Why / Context
@saddlepaddle, tagging you since #5964 was yours and its objections shaped this design.
We currently ship zero performance telemetry from the desktop app. This week a user publicly churned over freezes and memory ("Superset froze my 64GB MacBook"), and reconstructing what happened took forensic work against the residue of old 1.16/1.17 events. Their Electron RSS never passed 2GB; the likely culprit (the pre-#6352 WebGL glyph-atlas growth, and whatever their agent child processes were doing) lives entirely in dimensions we never measured. We also cannot verify from data whether the 1.21.0 perf fixes actually moved the fleet, because every build after 1.17 reports nothing.
#5964's two objections, addressed head-on:
- Volume: the old sampler uploaded every raw sample, ~288 events/day/machine, projected ~1.4M/day on a ~1.3M/day baseline. The digest uploads 1 event/day/machine: at ~8k daily actives that is ~8k events/day, roughly 0.6% of baseline, and it scales with actives rather than with installed idle machines.
- Idle machines emitting all weekend: sampling contributes nothing once
powerMonitor.getSystemIdleTime()exceeds 30 minutes. An idle desktop goes silent.
How It Works
apps/desktop/src/main/lib/resource-digest/samples every ~5 min (jittered, unref'd): Electron per-process-class RSS viaapp.getAppMetrics(), child-tree totals + host memory/load via the existingresource-metricscollector (numeric totals only), main-process event-loop p99 viamonitorEventLoopDelay.- Samples accumulate in memory (bounded, a few KB); the digest flushes on UTC day rollover and on quit (partial days under 3 samples are dropped as noise).
- Threshold crossings (renderer >3GB, total >5GB, children >16GB, host memory >92%, event-loop p99 >1s, window unresponsive) emit an immediate
resource_alertwith a 6h per-kind cooldown. - Privacy unchanged from #5777: digest payloads pass a hard numeric-key allowlist; the only string an alert carries is its kind from a fixed enum. No IDs, paths, names, or commands can ride along.
- Same gates as before: off in dev/test, honors the telemetry-enabled flag,
track()additionally no-ops until a user is identified.
Manual QA Checklist
- Packaged build:
resource_digestarrives in PostHog after a day of use (telemetry is production-only, so this needs a canary/packaged run; not yet validated live) - Quit after >15 min of use flushes a partial digest with
flushed_on_quit: 1 - No sampling while machine idle (no digest sample growth overnight)
Testing
bun test src/main/lib/resource-digest/(8 tests: aggregation quantiles, day rollover, allowlist filtering, idle gating, quit-flush minimum, alert cooldown, unresponsive counting, disabled/idempotent start)- Mutation check: removing the idle gate makes the idle test fail, so the suite catches regressions there
bun run typecheck, Biome clean on touched files
Design Decisions
- Digest instead of raw samples: raw samples let PostHog compute arbitrary percentiles, but that flexibility is exactly what cost 2x ingestion. Per-machine daily p50/p90/max still answers every question we actually asked of the old data (fleet percentiles, release regressions, leak-vs-stable, worst-case tails).
- Reusing the
resource-metricscollector for child trees: it already computes per-workspace process-tree memory for the TopBar popover; the digest reads only the numeric totals. The 5-minute idle-mode call costs onepssnapshot, which the popover already does far more often when open.
Follow-ups
- Report the xterm WebGL glyph-atlas size from the renderer (the #6352 cap tracks page adds already); GPU memory is invisible to every RSS metric and was the likely mechanism in this week's churn case.
- Renderer JS heap in the digest (needs an IPC hop; kept out to stay main-process-only here).
- The Sentry
orgtag onhost-service crashedevents is empty in practice, so those 7.7k SIGABRT events/14d are unattributable; worth a separate fix.
Risks / Rollout
- Risk: low; the sampler never throws into app code (all sample errors are swallowed), timers are unref'd, and payloads are allowlist-filtered.
- Rollout: ships with the next desktop release; no server or flag dependency.
- Rollback: revert this commit; events stop at the next update, same as #5964.
Summary by cubic
Reintroduces desktop fleet resource telemetry as a low-volume daily digest and ensures quit-time events reach PostHog. Old behavior: no resource telemetry and quit-time captures could drop; new behavior: 5‑minute local sampling aggregates into one resource_digest per machine per UTC day plus cooldown‑limited resource_alerts, sampling pauses on idle machines, and the quit path drains analytics before forced exit.
- Core:
apps/desktop/src/main/lib/resource-digest/samples with jitter and aggregates p50/p90/max for Electron RSS (by class), child-process trees, host memory pressure, main-process event-loop delay, and counts window-unresponsive episodes. Flush on UTC day rollover; flush on quit only if ≥3 samples. - Alerts: thresholds for renderer/total/children RSS, host memory (>92%), event loop p99 (>1s), and window unresponsive; 6h per-kind cooldown.
- Privacy: numeric allowlist only; alerts carry
alert_kindand numeric sample fields; no IDs/paths/commands. - Gating: production-only, respects the telemetry flag, and
track()no-ops until a user is identified; timers are unref’d. - Wiring: start in
main/index.tsafter app setup; stop onbefore-quit.windows/main.tsrecords unresponsive episodes into the digest and sends a Sentry error. Quit sequence now awaits a time-bounded analytics drain (flushAnalytics, 2s) beforeforceExitto avoid losing quit-flushed events. - Tests: coverage for aggregation, idle gating, day rollover, quit-flush minimum, cooldowns, allowlist filtering, disabled/idempotent start, and quit-time analytics drain.
Rollout
- No migrations or server changes. Ships in the next desktop release. No action required.
Summary by CodeRabbit
-
New Features
- Added privacy-conscious resource monitoring to track performance and system health.
- Added daily resource summaries and threshold-based alerts.
- Added detection and reporting for periods when the main window becomes unresponsive.
- Monitoring respects telemetry settings, skips extended idle periods, and stops cleanly when the app quits.
- Added reliable analytics delivery during normal application shutdown, without delaying exit indefinitely.
-
Bug Fixes
- Improved shutdown handling when analytics delivery encounters delays or errors.