Add CI latency reports and an optional p95 regression guard

AJ
7d ago
+1736-39

Pull Request Checklist

  • I have read the CONTRIBUTING guide.
  • I have updated the documentation (if applicable).
  • My code follows the style guidelines of this project.
  • I have performed a self-review of my own code.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.

Description

Refs #435

The existing benchmark prints query timings locally. This change adds a GitHub
Actions workflow and a JSON report for the same fixed 100,000-document workload.
The runner builds a fresh local Moss index and records p50, p95, and p99 with
the commit and input details. It measures the installed SDK release.

The live job runs on trusted main runs when enabled. A reviewed p95 baseline
can turn a slowdown into a failing job. A separately captured reference can
report agreement between Moss and Pinecone, Qdrant, and Chroma on the same
text. Each engine handles its own embeddings. Low agreement is report-only.

This first stage covers the benchmark runner and reporting. Live Ubuntu execution,
latency calibration, a real competitor reference, and durable results storage
are open. The baseline and competitor checks
are opt-in. A calibrated baseline and a real reference are needed to activate
them. The current workflow retains result artifacts for 90 days.

The initial latency plan expanded to include the optional guard and agreement
report after I inspected the unlabeled corpus. A label source for relevance
recall is still open. Would competitor agreement be useful as an interim
quality signal for this workload?

Environment loading moves from import time into the legacy benchmark entry
points. The existing commands load their environment when invoked, while the
CI runner can import their shared functions without discovering a local
credentials file.

Validation

  • Before at 2e638ec, a search of .github/workflows found no call to the
    benchmark.
  • The local runner completed a fresh 100,000-document build and 750 measured
    searches on a Mac. The p50, p95, and p99 were 13.608, 32.357, and 75.283 ms.
    These measurements came from the first latency stage and are local evidence.
  • At 64494cb, all 24 offline tests passed. They cover a p95 regression that
    fails and keeps its report, and low competitor agreement that leaves the run
    successful. Ruff, Python compilation, actionlint, and the whitespace check
    passed.
  • Review caught an output path that could overwrite credentials or the corpus.
    The runner now rejects those paths, and the regression test verifies both
    inputs survive the rejected command.
  • Chroma and Qdrant returned five real results for all 15 queries on a local
    sample. A full three-engine capture requires a configured Pinecone index.
Before, on 2e638ec
$ rg -n 'benchmarks|run_all\.py|bench_moss' .github/workflows
no matches, exit 1

After
$ .venv/bin/python -m unittest discover -s benchmarks/tests -v
Ran 24 tests
OK

AI assistance

I used Codex to trace the existing benchmark, implement the runner and checks,
and help validate the results. I chose to keep the full corpus, leave generated
indexes out of Git, and compare the engines on the same text. I also used
Claude Code’s plan-check on the initial latency plan.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Summary by CodeRabbit

  • New Features
    • Added a benchmark workflow that runs checks on pushes and pull requests and can also be started manually. Live measurements run only under configured conditions, with results saved as an artifact.
    • Benchmark reports can include optional latency checks and comparisons with saved competitor rankings.
    • Added local benchmark runs using a fixed dataset, with result validation and repository and runtime details included in reports.
  • Documentation
    • Expanded benchmark guidance with setup, report details, workflow configuration, optional checks, and offline validation instructions.
  • Tests
    • Added offline tests for benchmark measurement, configuration, result reporting, and validation.