fix(pty-daemon): pause flooding PTYs on backpressure instead of destroying the shared socket

Merged
KI
4mo ago
+195-1

Links

  • Findings + design context: plans/20260717-host-service-git-worker-pool.md (#5746)

Summary

  • One full-rate flooding terminal (e.g. yes, a huge build log) used to trip the daemon's 8 MiB outbound cap and socket.destroy() the daemon↔host-service connection — severing every terminal in the org (host-service logs pty-daemon disconnected; closing N terminal WS socket(s); users see all terminals blip). Reproduced deterministically in the 2026-07-17 stress test.
  • Fix: per-session flow control. When a subscriber's socket buffers past 1 MiB, pause the producing PTYs (pty.pause()); resume on socket drain. The kernel PTY buffer fills and the flooding process blocks on its own write — it throttles itself at the source. The 8 MiB destroy stays as a last-resort for pathological control-message floods.

How It Works

  • Pty gains pause()/resume(): node-pty's public pause/resume (added upstream for VS Code's ptyHost flow control, same actuator) for spawned sessions, ReadStream.pause() for adopted-fd sessions.
  • Server.wireSession: after fanning out a chunk, any subscribed conn with writableLength > outboundPauseThreshold records the session in conn.pausedSessions and the PTY is paused. Paused PTYs emit no further data, so this fires once per congestion episode.
  • Resume paths: socket drain (guaranteed to fire — the buffer only passes the threshold after write() returned false), and conn drop (a destroyed socket never drains). A session held paused by another still-congested conn stays paused.
  • This is not a return of the removed ACK flow control (SUPER-939): no client involvement, no protocol change — the congestion signal is the daemon's own socket buffer. Renderer-slow congestion is still handled by host-service's existing bounded buffering; this covers the daemon→host-service hop.

Manual QA Checklist

  • Flood a terminal (yes) with other terminals open — no org-wide disconnect, other terminals stay live (covered by integration test with real shells + real unix socket; app-level re-run of the CDP flood stress is a follow-up)

Testing

  • New test/flow-control.test.ts (node --test, real /bin/sh + yes flood + real unix socket):
    • slow consumer → connection survives, output resumes after the consumer drains
    • consumer hard-disconnect while paused → next subscriber still gets output (conn-drop resume path)
    • Both proven to fail against the old behavior (mutation-tested: destroy-only → test 1 fails; no conn-drop resume → test 2 fails)
  • Full bun run test:integration (51/51) and bun test unit suite (57/57) pass
  • bun run typecheck (pty-daemon + host-service), bun run lint clean

Risks / Rollout

  • Risk: a paused PTY blocks the foreground process's writes — that's the intended semantics (same as VS Code), and pause windows are bounded by drain/disconnect. Sessions with no subscriber are never paused, so background jobs in unattached terminals are unaffected.
  • Rollback: revert; behavior returns to destroy-on-overflow.
Open in Stage

Summary by cubic

Prevent org-wide terminal disconnects by pausing flooding PTYs on backpressure instead of destroying the shared daemon socket. Adds per-session flow control to pty-daemon so heavy output (e.g., yes/build logs) doesn’t take down other terminals.

  • Bug Fixes
    • Implemented per-connection flow control: when a subscriber’s socket.writableLength exceeds a threshold (default 1 MiB via outboundPauseThreshold), pause the session’s PTY; resume on socket drain or connection drop. The 8 MiB outboundBufferCap remains a last-resort.
    • Extended Pty with pause()/resume() and wired both node-pty sessions and adopted-fd sessions.
    • Tracked pausedSessions per connection to avoid resuming while another subscriber is still congested.
    • Added test/flow-control.test.ts and included it in test:integration to cover slow consumer and disconnect-resume paths.

Summary by CodeRabbit

  • New Features

    • Added PTY flow-control support to pause output when clients cannot keep up and resume it when capacity is available.
    • Added configurable outbound pause thresholds for server connections.
    • Improved handling of disconnected clients so shared sessions remain available and recover correctly.

  • Tests

    • Added integration coverage for slow, interrupted, and reconnecting consumers during high-volume PTY output.