Make the ingress wait configurable and add an autoscaling guide

Merged
HA
6d ago
+465-20

Stacked on #268. Review and merge #268 first. This PR's base is feat/workers-default-231.

What changes

  • New setting SQL_FS_MAX_INGRESS_WAIT_MS. It sets how long a request or MCP listener waits for ingress capacity before it gets a 503 EWORKER_BUSY. Before, the wait was a hard-coded 250 ms. The default stays 250 ms, so nothing changes unless an operator sets it. It is read the same way as the other SQL_FS_MAX_INGRESS_* settings.
  • New guide: docs/AUTOSCALING.md.
    • The /healthz, /readyz and /metrics endpoints, and how a scraper authenticates to /metrics.
    • A reference for every metric /metrics emits. The names were captured from a running server's /metrics output and checked against metrics.ts.
    • Signals to scale on, ranked: the refusal rate first, then queue depth and the worker wait p95 as early warnings, then CPU and RSS. Each comes with example PromQL. The guide explains why the wait histogram tops out at the wait limit and misses refused requests.
    • What adding replicas does not fix: waits on one sandbox, locks and recovery, and the per-tenant limit on concurrent execs.
    • Measured capacity on a 2 GiB replica, and a rule of thumb for replica count.
    • Shutdown behavior: a SIGTERM closes ingress at once, running worker operations get 30 s, and the process force-exits at 60 s.
    • An Azure Container Apps example matching deploy/azure/aca.yaml.
  • Updated docs. docs/CONFIGURATION.md gets the new row and names both wait settings, and the autoscaling list there now points to the new guide. README.md and AGENTS.md link to it.

Why the default stays 250 ms

We planned a 1000 ms default, then measured it. One replica with 2 GiB of memory and 2 CPUs, and clients that waited 1 s after each 503 as Retry-After: 1 asks:

Load250 ms1000 ms
100 agents start at once, 10 tenants83 refusals83 refusals
One tenant, about 100 execs/s offered, three runs: refusals317, 385, 395403, 479, 414
The same runs: p99 latency123, 151, 174 ms246, 283, 173 ms

The ingress queue holds no more requests than ingress admits at once (maxQueued: options.maxRequests, 8 per replica and 4 per tenant). A longer wait therefore cannot absorb a burst: everything beyond those 8 is refused at once. #269 tracks a deeper ingress queue. The guide's "Wait limits" section has the table.

Notes for reviewers

  • deploy/azure/aca.yaml is unchanged. The guide explains that its scale rule of 50 concurrent requests per replica fires long after 503s begin, because one replica admits 8. Fixing the file is left for a separate change.
  • Neither shutdown limit (30 s and 60 s) can be configured, while a script can run for 120 s by default. The guide tells operators to set a termination grace period above 60 s.
  • The Container Apps YAML and the PromQL examples have not been run against a real platform or Prometheus.

Tests

  • A request and an MCP listener queued with SQL_FS_MAX_INGRESS_WAIT_MS=40 are still queued at 39 ms. At 40 ms each gets a 503 EWORKER_BUSY with Retry-After: 1. These tests fail if either gate ignores the setting.
  • The default is 250 ms even when SQL_FS_WORKER_QUEUE_WAIT_MS is 900. An explicit value wins.
  • Invalid values, such as 0, -1, 1.5 and NaN, stop startup.

Verification

  • pnpm check passed: 2332 and 154 tests.
  • pnpm verify:local passed: contracts 94, output-cap 26, integration 265 (none skipped), SDK clients 102.

Summary by cubic

Adds SQL_FS_MAX_INGRESS_WAIT_MS to control how long a request or MCP listener waits for ingress capacity before a 503 EWORKER_BUSY, fixes shutdown hanging on open MCP listener streams, and adds an autoscaling guide. The default stays the previous hard-coded 250 ms; measurements showed a longer wait cannot absorb bursts because the ingress queue caps at 8 requests per replica.

  • Shutdown now ends GET /mcp listener streams first, so a client holding one open no longer blocks the drain until the 60 s force exit. In-flight MCP tool calls still finish, and terminating replicas refuse new connections. The new scripts/verify/shutdown.mjs asserts that SIGTERM with a listener open ends the stream cleanly and exits before the force-exit timer, and scripts/verify-local.sh runs it.
  • Both SQL_FS_MAX_INGRESS_WAIT_MS and SQL_FS_WORKER_QUEUE_WAIT_MS now reject values above 2147483647 ms, which Node's timer would otherwise run after 1 ms. Invalid values such as 0, -1, 1.5 and NaN already stopped startup.
  • docs/AUTOSCALING.md is new and covers health and metrics endpoints, every exported metric, ranked scaling signals with example PromQL, measured single-replica capacity, shutdown draining, and an Azure Container Apps scale rule. It also lists the settings a replica needs when running 4 workers. docs/CONFIGURATION.md, README.md, and AGENTS.md link to it, and docs/CONFIGURATION.md adds the setting row.
  • CI and Python SDK CI now run on push to main only; the old branch push patterns ran each workflow twice per push.