Add the desktop benchmark harness with a frozen artifact protocol

Draft
CO
2mo ago
+1742-0

Summary

Problem: ANLG-320 Phase 0 starts with "implement the benchmark harness and freeze the protocol before optimization or UI porting", and the stop conditions after Phase 0 and Phase 3 depend on numbers from it. Nothing measures whole-application CPU and memory across the process tree (Tauri's WebKit helpers, sidecars) in a reproducible, versioned format, so the existing 1.4.19 evidence in the issue remains an uncontrolled snapshot.

Fix: Add crates/desktop-bench (anarlog-bench), a two-command harness:

  • run launches (or attaches to) a build, samples the entire process tree at a fixed interval, and writes one schema-v1 JSON artifact per trial. The process set is the root, its transitive descendants, and any process whose name matches --include-name and started at or after launch, which is how reparented WebKit content/network processes and sidecars get attributed (the issue explicitly warns parent PID alone is not enough). Every process ever seen is recorded with pid, parent, command line, start time, and the rule that matched it, so a run's process set can be audited. Per sample: CPU (100 = one logical core), memory (Linux PSS from smaps_rollup so summing over processes does not double count shared pages; RSS elsewhere, with RSS always recorded alongside), disk deltas, thread count. Summaries: distributions with median/p95/cv, CPU core-seconds, memory plateau and least-squares growth after a --plateau-after offset (the soak gate's input). --warmup-trials/--trials implement the 1 + 5 procedure; --meta key=value records power/display/thermal/locale/CloudSync state and warns when the protocol's required keys are missing.
  • report loads artifacts, refuses to mix schema versions, excludes warm-ups, and renders a Markdown table per (scenario, fixture) with per-runtime medians across trials, cv, and deltas against a baseline, plus the build SHAs and memory metric per runtime so a mixed or debug comparison is visible on the page.

PROTOCOL.md is the frozen operational protocol: field definitions, process-set rules, per-platform memory metric definitions, summary math, and an explicit list of what v1 does not measure (launch-to-interactive, input-to-paint, render/invalidation counts, IPC/SQLite/CloudSync counters, GPU/energy), which need the Phase 2 in-app hooks and a schema bump. Any change to the definitions requires bumping SCHEMA_VERSION.

Decisions worth a look:

  • Built on sysinfo (already a workspace dependency); threads are filtered out of the process walk because with_tasks() lists them as processes on Linux.
  • macOS records RSS, not phys_footprint, until a proc_pid_rusage path is added; PROTOCOL.md flags macOS memory as an upper bound. The issue asks for physical footprint, so this is the first follow-up.
  • The harness excludes its own pid from name matching. Without that, --include-name anarlog selected anarlog-bench itself and the end-of-duration kill took the harness down before it wrote the artifact (found during the smoke run below).
  • CI: clippy (--no-deps -D warnings) and tests added to the Linux desktop job; the macOS workspace test picks the crate up automatically.

Verification

  • cargo clippy --locked -p desktop-bench --all-targets --no-deps -- -D warnings
  • cargo test --locked -p desktop-bench (19 tests: stats, summary derivation, JSON round-trip, process-set selection incl. reparented helpers/cycles/self-exclusion, PSS parsing, report aggregation and schema rejection, CLI parsing)
  • cargo metadata --locked; pnpm exec dprint check on changed files
  • Smoke on Linux: run against bash -c 'yes > /dev/null & sleep 30' with --warmup-trials 1 --trials 2 --duration 4 captured 3 processes (root + 2 descendants), CPU median 99.5%, PSS 0.85 MiB vs summed RSS 7.8 MiB, and killed the tree cleanly; run against the GPUI shell from #7356 for 6 s captured the startup spike (103% at t=1s) then 0% idle at a 95.8 MiB PSS plateau with 24 threads; report --baseline tauri rendered both runtimes with deltas.
  • Attached (--pid) to the running Tauri debug dev build (turbo dev:desktop on Linux/WebKitGTK) for 20 s with --include-name WebKit: the process set recorded the desktop root plus WebKitNetworkProcess and WebKitWebProcess, 60 threads, 960 MiB PSS, 3% idle CPU. Debug numbers are not comparable per the protocol; this validates attribution on a real Tauri tree.
  • Not run: macOS/Windows.
Open in Web Open in Cursor