Add CI performance benchmark suite

MA
2mo ago
+8603-215

Summary

Add a reproducible two-tier performance benchmark suite for detecting
regressions in pull requests.

  • Tier 1 runs pinned lmbench workloads for syscall, process, and
    filesystem latency.
  • Tier 2 measures Python startup, Git status, ripgrep, zstd, and Make
    against a deterministic committed corpus.
  • Results include medians and raw samples in structured JSON.
  • Baseline columns are prepared for elfuse, QEMU-based native Linux,
    and OrbStack.

CI integration

Add a benchmark leg to the self-hosted Apple Silicon runtime matrix.
The leg runs exclusively on the host to avoid interference from sibling
jobs, compares results in report-only mode, and uploads the JSON artifact.

Missing baseline metrics are treated as failures, preventing broken or
timed-out benchmarks from silently passing. Benchmark preparation and
cleanup commands are also bounded by timeouts.

The checked-in baseline has not yet been captured on the designated
self-hosted CI machine. It will be refreshed in a follow-up after the
benchmark leg is deployed and runner variance has been characterized.
Comparison therefore remains report-only for now.

Validation

  • Full cold run: approximately 10m 19s
  • Cached CI-equivalent run: approximately 9m 20s
  • Baseline match exits successfully
  • An injected 2x regression exits non-zero
  • A missing baseline metric exits non-zero
  • Benchmark output contains 9 lmbench and 6 application metrics

Closes #195

Summary by cubic

Adds a two-tier performance benchmark suite with a deterministic workload to catch syscall, process-startup, and app-level regressions. Runs on the self-hosted Apple Silicon runner under an exclusive host lock and posts a sanitized PR comment with benchmark deltas. (Issue #195)

  • New Features

    • tests/bench-suite.sh: Tier 1 lmbench (lat_syscall/lat_proc/lat_fs) and Tier 2 app workloads (git status, rg, make, zstd) with warmups; guest-side timing for qemu-aarch64 via tests/bench-timeit.c; writes medians+samples to build/bench-results*.json.
    • Deterministic tests/bench-corpus for stable Tier 2 inputs; excluded from formatting to preserve baselines. Fixtures fetch app tools and cross-compile pinned lmbench.
    • Baselines in tests/bench-baseline.json for elfuse-aarch64, qemu-aarch64, and orbstack. scripts/bench-compare.py flags regressions (tunable via BENCH_REGRESSION_THRESHOLD) and treats missing metrics as NEW by default; scripts/bench-promote.py refreshes baselines.
    • CI: make bench / make bench-ci; BENCH_ENV selects elfuse-aarch64 (default) | qemu-aarch64 | orbstack | native. Benchmark leg isolates measurements with scripts/ci-host-lock.sh. .github/workflows/bench-report.yml uses scripts/bench-report.py to publish a sanitized advisory PR comment from the uploaded report artifact. Docs in docs/testing.md.

  • Migration

    • Run: make bench (or BENCH_ENV=qemu-aarch64 make bench-ci).
    • Compare: scripts/bench-compare.py --report-only.
    • Refresh baselines: run make bench-ci per env, then scripts/bench-promote.py; commit the updated tests/bench-baseline.json.