benches: warm, two-quick-presses and cold arms in six felt benches; the headline is the worst arm

LA
7d ago
+774-71

Summary

Reward-hacking audit 2 item 3c: six felt benches waited 1-120 s for made-ahead work (the spare New Tab page, the spare Chromium window, the Ctrl-` stand-in, the icon picker warm-up, the agent pool) before the timed press. So they never timed what a person waits for when they press quickly or right after launch.

Each of cef-tab-bench.py, cmd-n-paint-e2e.py, new-workspace-real-typing-e2e.py, bench-icon-picker-open.py, agent-pane-live.py and instant-new-surfaces-bench.py now times three arms of its press (shared code: scripts/cmux-next/bench_arms.py):

  • warm: the existing runs, after the made-ahead work is ready (unchanged);
  • quick: two quick presses; the timed press comes right after the first, with no wait for a refill;
  • cold: right after launch, or right after the spare was dropped, with no warm-up wait.

Every arm prints n, missed, p50, p95, p99 and max. The bench's HEADLINE is the WORST arm (a defect or a missed run first, else the highest p95, then the highest max). In cmd-n-paint the defect is a blank frame: an arm with any blank frame is the HEADLINE, and the HEADLINE names its blank_frames count. Each script prints that rule with its HEADLINE line and states it in its doc text.

No CI gate is added: instant-new-surfaces-bench.py --budget-frames still reads only the warm rows, and cmd-n-paint-e2e.py keeps the exit status of its warm (and --cold launch) openings; its quick and cold openings are reported only.

The intentPrewarmPress press-turn timing and the false doc comment in IntentPrewarmInputs.swift are on perf/chrome-parity only (that code is not on this branch): cp/fix-bench-worst-arm.

Performance

No perf numbers: this PR changes bench scripts only (no app code), so it has no before/after of its own. The numbers the new arms measured are in the PR comments (instant-new-surfaces 6520438640, cmd-n-paint and icon-picker 6520956845, the perf branch's prewarm-press 6521478443).

Changelog

none

Testing

  • python3 -m unittest scripts/cmux-next/tests/test_bench_arms.py: 6 pass (Linux VM).
  • Fleet runs of the changed benches: in the comments below, with each job id and the numbers per arm.

Observed / Not run

  • Observed: the unit tests above (7 pass); py_compile of the six scripts; the prepush gate (no new violations).
  • cef-tab: not run (the controller runs an unlisted scripts/measure script only from feat or main; HTTP 400). It runs as the first step after this PR lands, and its numbers go on this PR.
  • Not run yet: the fleet runs of instant-new-surfaces-bench, cmd-n-paint-e2e and bench-icon-picker-open. The controller's state disk was full on 2026-10-30 from 09:2xZ (submits timed out). Their job ids and the numbers per arm come in a comment.
  • NOT RUN: agent-pane-live.py. It needs a GUI host with signed-in claude and codex harnesses (nx-remote on cmux-lawrence-2). This VM has no nx-remote, and the controller has no step for it. Its arms are code-reviewed only.
  • NOT RUN: new-workspace-real-typing-e2e.py. It must activate the app and post real HID key events (--needs input). A capture slot refuses activation, and the controller has no step for it. Its arms are code-reviewed only.