fix: bound idle connections in decision server

SH
7h ago
+110-9

A client that leaves headers or a declared body unfinished can hold the local decision server in a blocking read, preventing queued health requests and graceful shutdown.

Use nonblocking connections with absolute budgets of 5 seconds for headers and 30 seconds for bodies. Response writes share one absolute budget: 2 seconds plus 1 second per started MiB, capped at 30 seconds. Progress does not renew a deadline. Idle operations poll shutdown every 20 ms; writes check between 64 KiB chunks. Request timeouts receive a best-effort 408. Model access stays serial, and existing size limits and validation remain intact.

Review follow-up

  • Fixed large-response truncation with a size-dependent, capped write deadline. A reader paused for 2.5 seconds received a complete 4 MiB body; the old writer cut the same response short. A fast reader received all 32 MiB.
  • Nonreading and trickling readers reach the 30-second cap without renewing it; a write interrupt exits in under 0.5 seconds. All 14 request-transport probes, eight budget boundary checks, formatting, and whitespace checks pass. These are transport checks without a loaded model.
  • The fresh local serve-test rebuild was stopped under memory pressure before tests ran. Full-crate verification for the updated commit remains pending hosted CI; the results below describe the original implementation.

Validation

  • Windows x64, pinned kit 0.20.43: release product build, model-free smoke checks, all five existing serve tests, formatting, contract/version checks, and test ledger pass. The local build required newer runtime libraries and an existing import library; its service origins are reserved for validation.
  • Baseline socket reproduction blocked health requests and interrupt handling until the client disconnected. All 16 fixed socket probes pass, covering idle/partial/trickling requests, HTTP errors, non-reading clients, and interrupts. Health recovered after approximately 5 or 30 seconds; socket-interrupt exits were under 0.25 seconds. A separate reader received a complete 32 MiB synthetic response. These probes compile the actual pre-inference transport code and do not load a model.
  • Full suite: 4/6 checks pass, with 835 Rust tests passed, 3 failed, 8 ignored. Two failures require Windows symlink privileges; a streaming keepalive failure passes an isolated repeat without changes; a local-harness check fails because a temporary configuration directory is present. The original run remains non-green, and the streaming failure's cause is unproven.
  • Strict lint reports unused imports/items and redundant formatting in unchanged files. The instruction-file symlink check also fails with this checkout's symlink settings. No full unmodified-main comparison was run.

Model-backed scoring, shutdown during inference, and Unix execution remain unverified. Socket deadlines do not cancel synchronous inference. Repeated clients can still consume serial service time; this bounds each connection rather than adding concurrent admission.

Summary by cubic

Fixes idle connections in the decision server: a client that leaves headers or a declared body unfinished can no longer hold the server in a blocking read, which previously stalled queued health requests and graceful shutdown. Connections are now nonblocking with absolute deadlines — 5 seconds for headers, 30 seconds for bodies, and 2 seconds plus 1 MiB/s for response writes, capped at 30 seconds — and progress does not renew a deadline. Idle reads poll shutdown every 20 ms; writes check between 64 KiB chunks. Timeouts receive a best-effort 408, and requests interrupted by shutdown are skipped. Model access stays serial, and existing size limits and validation remain intact. Deadlines do not cancel synchronous inference, and repeated clients can still consume serial service time; this bounds each connection rather than adding concurrent admission.

Validation

  • Windows x64 passes the product build, smoke checks, all serve tests, and 16 socket probes covering idle, partial, trickling, HTTP-error, non-reading, and interrupt cases.
  • The full suite is 4/6 green: the three Rust failures are symlink privileges, a streaming keepalive failure that passes on isolated repeat, and a local-harness temporary-directory issue.
  • Linux, model-backed scoring, and shutdown during inference remain unverified.

Summary by CodeRabbit

  • Bug Fixes
    • Decision-server requests now time out if headers or request bodies take too long, returning an HTTP 408 response.
    • Requests interrupted by server shutdown are no longer scored, and connections that cannot be handled are skipped.
    • Responses that cannot be sent within the transfer deadline are cut short and the connection is closed.