measure: devtools-calls.sh, main-thread time of large Chromium DevTools calls in a real app (A/B)

LA
7d ago
+382-0

Summary

A fleet measurement for the CEF DevTools stalls (cmux-ad decision 5): the main-thread time of large Chromium DevTools calls in a real tagged app, as an A/B in one exclusive step. It is the live evidence #28916 and #28917 lack (their numbers are from a test rig).

  • scripts/measure/devtools-calls.sh: each arm is a commit (--arg=arms=base:<sha>,head:<sha>), built in the step as its own tagged app (bench configuration: Release optimization with the debug.* methods kept) in a worktree, then measured interleaved on the same host. The arms are built inside the step because the worker runs a measure script only at a commit whose copy of the script is the reviewed one, and a base commit has no copy.
  • scripts/cmux-next/devtools-calls-bench.py: launches each app's binary directly (scratch HOME, CMUX_NEXT_HANG_THRESHOLD_MS=8), opens a Chromium tab on a local page (no network), and sends Runtime.evaluate calls of 64 KiB, 512 KiB and 2.5 MiB through debug.cef.devtools (CEFTab.devTools, the path of an app page's script batch). Around each call it resets and reads debug.hangs: the longest main-thread gap of any length (max_gap_ms, so calls under the threshold are seen), and the longest stall record with its CPU time and whether its stack has a DevTools or JSON frame. It prints p50/p95/p99/max per arm and size (clean calls only) and writes every call to artifacts/devtools-calls.json. A run that stopped, or has a failed or missing call, prints DEVTOOLS-CALLS-INCOMPLETE <reasons> and exits 1; a whole run prints DEVTOOLS-CALLS-COMPLETE.

Not a CI job or gate: it runs only when someone submits it. No hq list entry is needed: the controller runs an unlisted scripts/measure/*.sh at a commit on feat-cmux-next or main (reviewedOnlyMeasure, cmuxterm-hq build-fleet/cmd/controller/ci_step.go:897; test TestAnUnlistedMeasureScriptRunsOnlyAtAReviewedCommit), and this script only runs at such a commit (it builds the arms itself). A list entry would only let it run at an unreviewed commit, which the A/B does not need.

Testing

Static only on this VM: bash -n, the bench's pure helpers (script size and escaping: a 2.5 MiB script is a 3.2 MB request, under the socket's 4 MiB line limit; percentiles; summary) checked with a stub. Not run: the script itself; its first run is the A/B after this lands.

Changelog

none

Proof

No capture: measurement tooling, no UI.