crash gate: a debug socket that never answers samples the app's main thread
When the debug socket never answers, the crash gate now says why: it samples the app's main thread and checks the socket in the app's state folder.
Base-red-job: cmux-next / cmux-next crash ends the app
What the red is
Every red run of the Crash ends the app step (cmux-scheme-compile) ends both launches with:
FAIL exception: the app's debug socket never answered (a Debug build is required); /tmp/cmux-debug-ci-crash-<run>-1.sock is absent, alive=True, last reply {'error': '[Errno 2] No such file or directory'}
The app's own output is only cmux: open-file limit 10240 -> 65536.
It follows the runner, not a commit
The suspected range (9e1c48fa22..e573965d34, including #28295 and #31904) is not the cause. The same failure shows up on owned minis hours before that range, and the same tips pass on Blacksmith macOS:
| run | built | runner | gate |
|---|---|---|---|
| 41267653860 | 05f7dc9a92 (11-01 03:14Z) | owned mini | FAIL, socket absent |
| 41284987256 | 6293c80da0 (11-01 09:10Z) | owned mini | PASS (socket answered after 31 s) |
| 41287191917 | 70ca1960a8 (11-01 09:08Z) | Blacksmith macOS | PASS |
| 41340639162, 41343175400, 41349600162, 41361139761 | 11-01 19:20Z to 23:16Z | owned minis | FAIL, socket absent |
| 41391725502 | 9efaeb7e8a (11-02 05:05Z) | owned mini | FAIL, socket absent |
| 41397849902 | 6924ccbb37 (11-02 06:29Z) | owned mini | FAIL, socket absent |
| 41395056515 | 418f3c73ae (11-02 06:55Z) | owned mini | FAIL, socket absent |
| 41396153603, 41386476436, 41403125592 | 11-02 04:10Z to 07:02Z | Blacksmith macOS | PASS |
Which commit looked like the first failure depended on whether that run's scheme compile landed on Blacksmith or on an owned mini. (#31600 keeps the landing DEV compile off Blacksmith macOS, which may be why owned minis now take more of these runs; I haven't verified that.)
The control socket opens only after settings.waitForLoad(atLeast: 1), the bounded launchSettle.settled(within: 4 s) and the classic-settings migrations (AppDelegate.startSettingsAndControl). On owned minis something there, or a blocked main thread, takes more than 180 s; the one passing owned run took 31 s. The log can't tell which.
This change
scripts/cmux-next/crash-ends-app-gate.py, on a socket that never answered:
- also reports
~/.local/state/cmux/cmux-debug-<tag>.sock, the real socket. The/tmppath is only its compatibility link, so an absent link alone doesn't show whether a listener exists. - samples the still-running app for 2 s, saves the report as
no-socket-<kind>.txtin the gate's out folder (already uploaded as thecmux-next-crash-gate-*artifact), and prints the main thread's call graph under the FAIL line.
The verdict is unchanged: still red. With the next red on an owned mini, the cause will be in the log.
Checks: py_compile; main_thread() tried on a synthetic sample report; scripts/cmux-next/prepush-gate.py reports no new violations.
Agent session: agent session
š¤ Generated with Claude Code
Summary by cubic
Improves the crash gate's failure report when the app's debug socket never answers, so the next red run names what kept the socket closed instead of only reporting an absent /tmp link. The gate verdict is unchanged: it still fails.
- Also checks the real socket in the app's state folder (
~/.local/state/cmux/cmux-debug-<tag>.sock), since the/tmppath is only a compatibility link to it. - Samples the still-running app for 2 s, saves the report as
no-socket-<kind>.txt, and prints the main thread's call graph under the FAIL line.