cmux NEXT: no main-thread waits behind the CEF map, bundle creation or link-menu LaunchServices; hang reports name the lock holder

Merged
LA
16d ago
+339-50

Summary

First-principles audit of cmux NEXT main-thread hangs, fixed at the cause. The evidence is Sentry hang_source:main_thread_watchdog events (14 days, project cmuxterm-macos) and 3 h of NIGHTLY AppKit layout logs.

Sentry / log signalCauseFix
MainThreadHang 2 s, __ulock_wait2 in dyld4::APIs::dlopen_from from WebKit Safe Browsing (issue 7787679434, 220 events), and dyld_image_header_containing_address from +[NSView clipsToBounds] (7791546153). Every event is 2-4 s after launch.The restored tab's CEF load maps the 367 MiB framework off main. dlopen holds dyld's loader lock for the whole map, about 2 s on a cold file cache (every nightly update; the log line "CEF ready ... load=2.1 s"). Every main-thread dlopen or dladdr waits for that lock.The library-load thread reads the binary into the file cache before cef_load_library. debug.cef prefetch and the "CEF ready" log now report map_ms (how long the lock is held).
MainThreadStall 2 s+ in _CFBundleGetBundleVersionForURL / getdirentries64 from DaemonRecoveryStrings.retryNow (7801916878)The first Bundle.module read in a module creates the bundle on main, and CFBundleCreate lists the bundle folder. 49 modules use bundle: .module.The launch warm-up thread opens all 55 resource bundles and loads their string tables.
Right-click spam (reported while a hang started)A right-click on a link or path in a terminal or agent chat asks LaunchServices for every app, the default app, and an icon for each app (up to 24), synchronously, on every click.LinkMenuAppCache keeps the app list per scheme or content type and refreshes it in the background. Icons are read once per app. The per-file default app is still asked each time (Get Info "Open with").
1564 "Unable to simultaneously satisfy constraints" in 3 h, up to 53/sHover card retarget: place() lays the glass out at the previous card's size, with required bottom and trailing edges. That is about 650 conflicts and repeats on every hover. The timeline document's bottom edge adds 204 more.Those edges now have priority 999. 999 still beats fitting-size compression, so fittingSize and the resting layout stay the same.

Guard (cheapest strong one for the class "main waits on a lock held by another thread"): the watchdog's hang report now also samples every other thread, right after the main thread. Threads are suspended one at a time, the walk writes to a preallocated buffer, and nothing allocates while a thread is suspended. The samples go to Sentry as threads, so a __ulock_wait2 hang names the thread that holds the lock.

Changelog

Fixed: cmux NEXT no longer freezes for about 2 s soon after launch when a restored browser tab starts Chromium, and right-clicking links no longer stalls.

Verification

Behavior tests at a product boundary are not added (repo rule: no unit tests). Regression evidence is in the field: after this lands, Sentry 7787679434 and 7791546153 should stop on new nightlies, and debug.cef prefetch.map_ms should be well below binary_ms on a cold launch.

Observed / Not run

  • Observed: scripts/cmux-next/check-concurrency.sh ok; scripts/cmux-next/prepush-gate.py no new violations; cmux-ci package-test-lane step (see comment) compiles CmuxNext with tests and runs MainThreadWatchdog*, HoverCard*, WorkspaceHoverCardSurface, BrowserProfileLinkMenu suites.
  • Not run: a tagged-app cold-cache launch measurement of map_ms; perf-gate GUI step; a live right-click burst timing. These are UNVERIFIED.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes cmux NEXT main-thread hangs from background loads and layout conflicts, and makes hang reports name the thread holding a lock a hung main thread waits on.

  • The CEF framework binary is read into the file cache before cef_load_library, so dyld's loader lock is held only for cached pages; the roughly 2 s stalls right after launch are gone.
  • All 55 resource bundles are opened and their string tables loaded on a dedicated warm-up thread (Bundle.warmResourceBundlesInBackground, moved to CmuxNextCompat), so a module's first Bundle.module read no longer runs on the main thread.
  • Link-menu LaunchServices answers and app icons are cached per scheme or content type and refreshed in the background; only http(s) links ask for app lists, and icons are read once per app.
  • Hover card and workspace timeline bottom/trailing constraints are now priority 999, removing the ~1564 constraint conflict logs in 3 h while keeping the same resting layout.

Diagnostics

  • Watchdog hang reports now sample every other thread right after the main thread — one at a time, into a preallocated buffer with no allocation while a thread is suspended — and send the samples as Sentry threads, capped to keep the event under Sentry's 1 MiB budget, so a main-thread __ulock_wait2 wait names its lock holder.