cmux NEXT: put the control socket back when its file is deleted under a running app (hq#1870)

Merged
TE
23d ago
+3208-713

Summary

If something deletes the control socket file while cmux NEXT is running, the app now binds a new socket at the same path. Until now, no client could connect again until a relaunch.

The readiness soak (hq#1870) hit this two nights running. On the soak host, a process outside the app removed /tmp/cmux-debug-<tag>.sock at about 11:00Z. The app kept listening on the unlinked inode, so every later connect got ENOENT. Agent turns kept completing because they go through acpmux, but the CLI and automation lost the app for the rest of the session.

How it works:

  • ControlSocketPathWatch watches the socket's directory with kqueue, with no timer. On each directory change it runs one lstat on the server's accept queue. It asks the server for a new listener only when nothing is at the path. Because the restore runs on the accept queue, it never races an accept that is pausing on descriptor exhaustion.
  • The new socket is bound the same way as the first one: in a private staging directory, with its mode set, then renamex_np(..., RENAME_EXCL). A path that another server or a file took in the meantime is never replaced.
  • The new listener is resumed before it is published. If a stop lands mid-restore, the new listener is resumed, cancelled, and its file removed.
  • A restore that keeps failing would trigger itself again, because its bind stages a directory beside the path. After 3 failures in a row, the watch ignores events and retries on a DemandTimer with Backoff from 1 s to 60 s until a restore works.
  • stop still unlinks only this server's own inode, and connections that were open across the delete keep working.
  • The socket-file helpers move unchanged into ControlSocketServer+File.swift, which keeps the server file under its 400-line budget. The stale-socket probe now sets SO_NOSIGPIPE, so the crash-safety check passes for that file.
  • plans/cmux-next/idle-wakeups.md gets a row for the new wake source.

Testing

New suite ControlSocketPathRestoreTests, driven through the real Unix socket on the fleet (run-swift-testing-suites.sh Packages/macOS/CmuxNext, Xcode 26.6):

  • aDeletedSocketFileComesBackWithItsMode: deletes the file under a running server and expects:

    • a new connect gets PONG within 5 s;
    • the file is back as a socket with the access mode's permissions;
    • an open connection keeps working.

    It then deletes the file again, and finally checks that stop removes it.

  • aPathAnotherServerTookIsLeftAlone: a plain listener's socket is renamed over the path. The test expects the path to keep that inode, including after the first server stops.

  • aStopDuringARestoreClosesTheNewSocket: the restore's bind calls stop().

    • Red at fd614cbaf254: the test process crashed in libdispatch (a never-resumed source was cancelled and released).
    • Green.

  • restoresThatKeepFailingBackOffAndStillComeBack: binds fail after the delete. The test expects at most 12 attempts in 2 s, then a PONG once binds work again.

    • Red at 1e0435647714: the watch had stopped for good, so there was no PONG within 10 s.
    • Green at 8dd3953bf70f: 3.3 s, and every suite passed.

Local checks: check-concurrency.sh, check-crash-safety.sh, crash_ratchet.py and check-no-godfiles.sh show nothing new.

Changelog

Fixed: cmux NEXT's control socket comes back on its own when something deletes the socket file, so the CLI reconnects without relaunching the app

Proof

No UI. The socket-level tests above are the proof.

🤖 Generated with Claude Code

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Summary by cubic

If the control socket file is deleted while cmux NEXT is running, the app now rebinds a socket at the same path instead of keeping the listener on the unlinked inode, where no client could connect until a relaunch. The readiness soak (hq#1870) hit this when an external /tmp sweep removed /tmp/cmux-debug-<tag>.sock.

  • ControlSocketPathWatch watches the socket's directory with kqueue (no timer) and runs its check inline on the accept queue, so a restore never races an accept pausing on descriptor exhaustion.
  • It asks the server for a new listener only when nothing is at the path; the socket is bound like the first (staging dir, mode, renamex_np(..., RENAME_EXCL)), so a path another server or a file took is never replaced.
  • The new listener is resumed before it is published; a stop landing mid-restore cancels it and removes its file. After three failed restores in a row the watch ignores events and retries with a backoff capped at 60 s until one works. stop still unlinks only this server's own inode, and connections open across the delete keep working.
  • Socket-file helpers moved to ControlSocketServer+File.swift; the stale-socket probe sets SO_NOSIGPIPE as the crash-safety check requires.
  • New ControlSocketPathRestoreTests drive the real Unix socket: restoring the file with its mode, leaving a path another server took alone, closing a new socket when a stop lands mid-restore, and backing off when restores keep failing. The test client stores its descriptor only after a connect succeeds, so a failed connect's deinit never closes a descriptor another suite's listener holds.