perf(sm): supervise dialed connections without the read bridge

Merged
GO
16d ago
+2465-42

sm.Client watched each dialed connection through CloseNotify. On TCP and
TLS, the first CloseNotify call installs a reader bridge: a goroutine reads
the socket ahead of dispatch into a fresh 32 KiB buffer and hands each
chunk to the connection's reader over a channel. With dialed connections on
the protocol core, that bridge made dialed requests and watchdog exchanges
more than four times as slow as accepted ones at two processors.

Client supervision (the watchdog, Disconnect and the watchdog join) now
uses a private signal that the reader raises when a socket read fails for
good, before callbacks for messages already decoded have drained. Nothing
reads ahead. The core still waits for DispatchDone before it finishes,
accepted connections still stop supervision on DispatchDone, and SCTP
connections keep their transport's error notification.

CloseNotify keeps reading ahead for callers that ask for it. On a dialed
connection the first call now starts the bridge as soon as no socket read
is in progress, also from a running callback; an accepted connection still
starts it at its next read. The bridge reuses its buffers and builds the
errors.As target only when a read fails.

Without read-ahead, Client sees transport loss later, or differently, in
these cases, each covered by a TCP and a TLS test:

  • Peer loss while a sequential callback runs, or while the reader waits
    for concurrent dispatch capacity, is seen when the reader next reads the
    socket. Until then the watchdog keeps running and can log, report and
    send a DWR after the peer has closed. If the reader never resumes, only
    a write failure, Close, or a watchdog or Disconnect timer ends
    supervision.
  • Disconnect can return ErrDisconnectTimeout where the bridge reported
    ErrDisconnectClosed early, and a final DPA read before the EOF can now
    complete it.
  • A ReadTimeout deadline that expires while dispatch blocks the reader is
    replaced by the next read's deadline, as on accepted connections,
    instead of closing the connection.
  • A watchdog start or Disconnect that begins after the reader has already
    seen EOF observes the loss at once.

Calling CloseNotify restores early notification in each case.

BenchmarkInboundMessages, Go 1.26.9, 16 interleaved samples per case on a
shared host, median µs per exchange, main against this change:

Case1 P2 Ps
Dialed/ApplicationRequest32.8 → 26.0111.9 → 25.3
Dialed/ApplicationAnswer25.1 → 18.192.9 → 77.2
Dialed/WatchdogExchange36.9 → 31.0125.8 → 30.9

Dialed exchanges allocate 32 KiB less, in two fewer allocations. Accepted
exchanges show no significant change. With an explicit CloseNotify
subscriber the bridge saves the same memory and is 17–28% faster at one
processor. At two processors its answer exchange is 8% slower (92.1 →
99.5 µs) and the other two cases show no significant change. Restoring the
per-read allocation removes that slowdown and a per-connection buffer pair
does not, so the bridge keeps the shared pool.

Closes #134.


Summary by cubic

Supervises dialed connections on sm.Client through the reader's own terminal signal instead of the CloseNotify read-ahead bridge, making dialed exchanges up to four times faster at two processors. Previously, the first CloseNotify call on a dialed connection started a goroutine that read the socket one chunk ahead into a 32 KiB buffer, which slowed dialed requests and watchdog exchanges more than fourfold. Supervision now observes peer loss when the reader next fails a socket read, without reading ahead.

Behavior changes

  • Peer loss while a sequential callback runs, or while dispatch capacity is occupied, is seen only when the reader next reads; watchdog timers and Disconnect deadlines remain active until then.
  • Disconnect can return ErrDisconnectTimeout instead of ErrDisconnectClosed, and a final DPA read before EOF can now complete it.
  • A read deadline that expires while dispatch blocks the reader is replaced by the next read's deadline, matching accepted connections.

Calling CloseNotify still installs read-ahead for callers that want early notification, and accepted connection behavior is unchanged. Benchmarks show dialed application requests dropping from 111.9 µs to 25.3 µs per exchange at two processors, with 32 KiB less allocation per exchange.