docs(benchmarking): drop the 3% default band; a gain is outside the declared variation criterion

Merged
BO
3d ago
+90-23

hr-benchmarking no longer has a 3% default band for "no clear change". There is no minimum gain size: a difference counts as a gain when it is outside the variation criterion declared before the first run (the comparison's own run-to-run spread) and the outputs are identical.

Before, the skill said to use 3% as the default no-clear-change band and to widen it only when the declared variation threshold was larger. That made a real, repeatable gain smaller than 3% read as "no clear change" even when the comparison's own spread was far smaller. The criterion each comparison already declares before its first run is now the only line.

What changes:

  • A gap inside the declared variation criterion is still reported as no clear change, with the statistic and threshold named beside it. A gap outside it, with identical outputs, is a gain.
  • Memory and CPU costs are reported beside the gain.
  • If the criterion cannot support a percentage, the result stays separate and unsupported differences are inconclusive (unchanged).
  • hr-writing-pr: the benefit chart script no longer defaults band to 0.03. The spec must give band, the declared criterion as a fraction of the baseline; a spec without one now exits with a message. The skill text, thread-visuals.md and the upstream-contributions rule say "declared variation criterion" instead of "reporting band".

Kept, on purpose: rules/core.md and hr-what-next/references/reporting.md ("label changes inside the noise band 'no clear change'") carry no number, and "noise band" still means the comparison's measured variation. The "skip the chart when the one row moved by less than 10%" rule in hr-writing-pr is a chart-size choice, not an acceptance threshold. The evals/end-of-reply/screen/benefit-column.md fixture mentions "noise band about 3%" in a tool result, which is test data. The CHANGELOG.md line about the 3% band under 0.11.0 is history.

Checks: bun test skills/hr-writing-pr (8 pass, including a new test for the required band); node --test on the composer, reference-link and rule-reference tests (46 pass); pinned oxfmt --check and oxlint clean. The full suites run in CI. Not landed; the maintainer decides. A CHANGELOG.md entry is under Unreleased.

Summary by cubic

hr-benchmarking no longer has a 3% default band for "no clear change". A favorable difference counts as a gain when it lies outside the variation criterion declared before the first run and the outputs are identical; an unfavorable one outside it is a regression labeled worsening. A change at or inside the criterion is still no clear change, with the statistic and threshold named beside the result, and memory and CPU costs are reported beside the gain.

  • hr-writing-pr's benefit chart script now requires the spec's band (the declared criterion as a fraction of the baseline, a finite non-negative number) and exits with a message when it is missing or invalid, including booleans, instead of defaulting to 0.03. When the criterion cannot be a fraction of the baseline, the chart is skipped and the result labeled inconclusive.
  • Skill text, thread-visuals.md, and the upstream-contributions rule now reference the declared variation criterion instead of a reporting band, and thread-visuals.md also requires identical outputs.
  • Changelog entry added under Unreleased.

Summary by CodeRabbit

  • New Features
    • Benchmark reports classify favorable results outside the predeclared variation criterion as gains when outputs are identical, and similarly unfavorable results as regressions. Differences within or equal to the criterion are reported as “no clear change.”
    • Reports show memory and CPU costs alongside gains and mark unsupported percentage comparisons as inconclusive.
  • Documentation
    • Benefit-chart specifications require a finite, non-negative variation band; there is no default band or minimum gain size. Charts are skipped when the criterion cannot be represented as a band, and invalid or missing values produce an error. Guidance consistently refers to the declared variation criterion.