perf(cef): build a large DevTools call's JSON off the main thread, in its browser's call order
Summary
Lawrence's hang records show the main thread busy for up to 1.5 s while cmux builds the JSON of one Chromium DevTools call:
| Record | Stall | Main-thread CPU | Top frames |
|---|---|---|---|
| v2.2.9 seq 538 | 566 ms | 563 ms | NSJSONSerialization dataWithJSONObject <- DevTools send <- CEFTab.devTools |
| v2.2.10 seq 356 | 1,563 ms | 1,044 ms | same |
| v2.2.9 seq 592 | 284 ms | 0.3 ms | same (wait inside the encode) |
On feat the same encode runs in CEFRuntime.devTools for every call. The large callers are an app page's script batches and replies (ChromiumPageHostBridge: up to 2 MiB of script per Runtime.evaluate) and an agent's Runtime.evaluate script on a fork before API 29. The v2.2.9/v2.2.10 builds carry perf's CEFRuntime.sendDevTools wrapper (748bda1361f1, not on feat), but the encode inside it is the same CEFDevToolsResult.params that feat calls on main.
Chrome's model: the UI thread never builds or parses large JSON (services/data_decoder); it only posts the message. This PR does the same:
CEFRuntime.devTools(_:method:payload:)(used byCEFTab.devToolsandCEFTab.evaluate's DevTools path) encodes parameters with more than 64 KiB of text on the cooperative pool (CEFDevToolsResult.encodedParams,@concurrent), then sends in a main-actor turn of its own.CEFDevToolsCallOrderkeeps each browser's calls in the order they were made. A large call takes its place in the caller's turn; a later call or freeze of the same browser goes after it, so a script never runs on a page frozen after it was called. A small call made when nothing of its browser waits still goes in the caller's turn, as before (the device-emulation synchronous-answer path is unchanged).- Freezes (
setFrozen,freezeNeverShownPage) go through the same order. CEFTab.evaluateparses its result off the main actor (parsedEvaluation), andCEFTab.snapshot/thumbnaildecode the screenshot answer off the main actor (decodedScreenshot), as the full-page capture already did.CEFDevToolsResult.paramschecksisValidJSONObjectfirst (JSONSerializationraises an Objective-C exception for a non-JSON value) and returns a native string.
The DevTools section moves from CEFRuntime+Events.swift to CEFRuntime+DevTools.swift.
The shim side of the same calls (CefParseJSON and the JSON rewrite inside cmux_shim_devtools_call, 61-104 ms in the same hang records) is a separate PR: cp/cef-shim-devtools-call-no-reparse.
Testing
Regression test: CEFDevToolsParamsOffMainTests (CmuxNextBrowserTests). It drives CEFTab.devTools, CEFTab.evaluate (fork API 28) and CEFTab.snapshot into a recording shim, while a main-actor ticker records the longest main-actor job (a main-thread blocking detector); budget 16 ms (one 60 Hz frame). Payloads: an 8 MB script with quotes, backslashes and non-ASCII text; a 3840x2400 noisy PNG answer. Two more tests check call order (a freeze and a small call made while a large call waits reach the shim after it, fork API 28 and 29) and that a small call still goes in its caller's first turn.
Focused command: cmux-ci run --class light --script scripts/ci/package-test-lane.sh --ref <sha> --arg=suite --arg=Packages/macOS/CmuxNext --arg=CEFDevToolsParamsOffMainTests
Red at b90eec6fec0 (test commit 1), fleet step d5952f7e5d3adbca5ca5ac9b (aws-m4pro-4): aLargeScriptCallNeverHoldsTheMainActorForAFrame fails, 6 MB script held the main actor 34.3 ms; anAgentsLargeEvaluateNeverHoldsTheMainActorForAFrame fails, 22.1 ms (budget 16 ms). The two order tests pass on the base (the base sends everything in the caller's turn, so it keeps order).
Red at 6e3cea6d621 (test commit 2), step 249be16dedf14a56099c4b2d: aLargeScreenshotAnswerIsDecodedOffTheMainActor fails, a 40 MB screenshot answer held the main actor 69.6 ms; the other 4 pass.
Green at 61e821c8e36 (head), step 5bbbce042ada3ab061b9159d (aws-m4pro-4): CEFDevToolsParamsOffMainTests 5 tests (6 cases) pass, plus CEFRendererPathsTests, CEFAgentRelayDeliveryCostTests, CEFDeferredTabShowTests, CEFQuitCreationTests, CEFDeferredFocusTests, CEFExtensionStoreReadTests: 7 suites, 36 tests, 0 failures.
Compile: cmux-ci compile-check --package CmuxNext --app --recipe-version 2 job be8035d54533ae2305dc919f passed at the head (CmuxNext 283 s, app 64 s, cmux-tui 36 s). Test targets: package-test-lane.sh compile-tests 7ebb41b3e03 step 54c0e5034c01a55d78932b90 at the head: every planned test target compiled (7).
Observed / Not run
Observed: the red and green steps above; the main-actor times are from the test's own ticker on a fleet Mac (aws-m4pro-4), not from an app.
Not run: a live before/after in a real app. I prepared a capture run (cmux-ci capture with steps that open a Chromium tab, send 64 KB, 512 KB and 2.5 MB Runtime.evaluate calls through debug.cef.devtools and read debug.hangs). It did not run: every capture mini refused this VM's SSH probe ("Host key verification failed"), and the Blacksmith fallback needs glaeda-gh, which this VM does not have. The journeys and Austin's-day runs were not run for this PR.
Performance
Live, in a real app (scripts/measure/devtools-calls.sh): the longest main-thread gap during one Runtime.evaluate through CEFTab.devTools, 60 calls per cell, bench-configuration apps built and measured interleaved on one host. Three jobs on two hosts agree; full distributions and 95 % bootstrap intervals in the PR comments of 2026-10-31.
| metric | before | after | delta | harness or run link |
|---|---|---|---|---|
| 2.5 MiB call, main-thread gap p50 (this PR alone: shim arm vs both) | 19.70 ms | 13.29 ms | -6.41 ms [-6.49, -6.37] | devtools-calls.sh job 9de334c59230ef9b2513af43 (cmux18) |
| same, second host | 19.75 ms | 13.34 ms | -6.41 ms [-6.61, -6.34] | job d920125ebc82a3d582f13567 (cmux24) |
| 2.5 MiB call p50, this PR alone against the old base | 38.45 ms | 31.81 ms | -6.63 ms [-7.04, -6.37] | job ec3e3def6b9823f7d708d24d (cmux18) |
| 2.5 MiB call p50, this PR with #28917, against neither | 38.36 ms | 13.29 ms | -65 % | job 9de334c59230ef9b2513af43 |
| 512 KiB call p50 (this PR alone) | 4.27 ms | 2.81 ms | -1.46 ms [-1.52, -1.42] | job 9de334c59230ef9b2513af43 |
| 64 KiB call p50 (this PR alone; small cost, follow-up) | 1.92 ms | 2.25 ms | +0.33 ms [+0.26, +0.39] | job 9de334c59230ef9b2513af43 |
| test rig: 6 MB script, longest main-actor job | 34.3 ms | under 16 ms | at least -18 ms | red step d5952f7e5d3adbca5ca5ac9b, green step 5bbbce042ada3ab061b9159d |
| test rig: 40 MB screenshot decode | 69.6 ms | under 16 ms | at least -53 ms | red step 249be16dedf14a56099c4b2d, green step 5bbbce042ada3ab061b9159d |
Changelog
Changed: A large DevTools call (an app page's update, an agent's script) holds the main thread less: its JSON is built off the main thread, and a large screenshot answer is decoded off it (a 2.5 MB script: 6 ms less alone, 38 ms to 13 ms with the shim change).
Proof
No capture: no UI change; the evidence is the main-thread time above.