fix: bound ingress queues and reduce replica startup contention

HA
11h ago
+2239-199

Summary

Related to #269.

Add independent ingress queue limits so replicas can absorb short bursts while ACA adds capacity. Preserve tenant fairness, byte budgets, execution limits and unknown-outcome safety.

  • Add SQL_FS_MAX_INGRESS_QUEUED_REQUESTS and SQL_FS_MAX_INGRESS_TENANT_QUEUED_REQUESTS. Zero disables waiting; defaults preserve the existing queue sizes.
  • Consume accepted queued bodies within existing byte budgets so client disconnects remove waiting requests. Once admitted, reads follow consumer demand. Immediate queue-full refusals leave the body unread.
  • Preserve completed and unknown outcomes when an ignored request body fails late. Never automatically replay uncertain executions.
  • Add ingress capacity gauges, outcome counters and admitted queue-wait histograms.
  • Record migration filenames and SHA-256 checksums under the existing transaction advisory lock. Unchanged replica boots skip filesystem DDL; pending DDL and history commit together.
  • Prune development dependencies from the runtime image, add reproducible ACA load/readback tooling, and update deployment and verification documentation.

Reviewer entry points

  1. src/api/ingress-admission.ts and src/api/worker-owner/admission.ts own queued input and admission lifetime. The queue, outcome and live ingress tests cover cancellation, input failures and cleanup.
  2. src/api/migrations.ts and its integration tests cover legacy adoption, changed checksums, rollback and concurrent startup while filesystem tables are busy.
  3. Dockerfile, deploy/azure/aca.yaml and docs/AUTOSCALING.md cover image pruning, scale settings and measured limits.
  4. scripts/verify/aca-load.mjs, its workload/barrier helpers and tests cover open-loop demand, safe busy retries, durable reconciliation and fixture cleanup.

Verification

Completed on 2026-10-18 against runtime source e0bf5f3; the final commit d486d7a changes only the measured-results documentation. All 32 owned tracked-file hashes still match the verified branch snapshot.

  • pnpm check: passed, including 2,358 core unit tests, 69 script tests, 154 TypeScript SDK tests and 174 Python tests.
  • pnpm verify:local: passed, including 272 integration tests and the standard live contracts and SDK checks.
  • pnpm verify:architecture: passed, including worker failures, SSE finalization, input budgets, ingress and tenant isolation.
  • Verification journeys: 41 mapped assertions plus 70 ingress checks passed.
  • Actual AMD64 image: Git, JavaScript, TypeScript, Python, SQLite, tar, MCP and cross-replica persistence passed.
  • Queued-upload regression: old code persisted abandoned input; the fixed artifact prevented all 25 eligible local trials, with positive controls. A separate real ACA abandonment proof also passed.
  • Migration regression: the old runner blocked on hot-table DDL; the final runner passed the corresponding live proofs.
  • pnpm security:audit: passed the configured high threshold. One existing moderate sprintf-js DoS advisory remains with no patched version.

Full gates ran in an isolated checkout of the exact candidate source because unrelated untracked user load-test files have existing lint failures. Those files are excluded from this PR. Current main merges cleanly with this branch; combined GitHub checks are separate from the retained candidate verification.

Real ACA results

Final immutable image:

ghcr.io/hazzng/sql-fs@sha256:fa390528895c1996b6ad0f3ccbc044b6d005c2d8125e9db51faeec506834f330
Final-image scenarioDurable resultClient p99
Ten ready replicas, 30 short writes/s for five minutes, no retries9,000/9,000 correct1.62 s
Cold scale-out from one to ten replicas, 12 operations/s for three minutes, each with 250 ms waiting work2,160/2,160 correct47.49 s including safe busy retries

The cold driver sent 3,533 attempts, including 1,373 explicit safe busy refusals before execution, within a 120-second operation deadline. Both final runs had zero unknown outcomes, generator drops or marker errors. These results do not establish the SDK default retry behavior or capacity for arbitrary production scripts.

The first retained new-pod durable work appeared after 28.36 seconds. Ten replicas were first observed ready after 46.40 seconds. These are observations from one episode, with ARM sampling bounds, not startup percentiles. Earlier exploratory and failed episodes remain distinguished in docs/AUTOSCALING.md.

All 45 ACA revisions were verified at zero physical replicas in two complete scans after testing. All owned fixtures were deleted and continuous test traffic stopped. Final development settings are min zero / max ten.

Rollout and limits

  • First adoption of migration history still applies the existing idempotent SQL files once. Keep traffic idle for that adoption. Applied-file checksum mismatches stop startup; no existing numbered migration was edited.
  • The ACA example scales early with native HTTP concurrency. A load-balanced /metrics scrape samples one replica and cannot supply a fleet aggregate by itself.
  • The tested app retained its existing disabled Redis read/write locks and blob cache. The deployment example retains the lock default; enabled-lock performance was not measured.
  • The throwaway control Redis has a measured 30-connection ceiling and uses volatile-lru; switching it to noeviction was rejected. Production requires noeviction, connection headroom for rolling overlap and longer tests matching the intended workload.
  • The image-size comparison in the documentation is an earlier same-source experiment. It does not demonstrate an ACA startup speedup.

The cold gap is covered by early scaling, available warm capacity, bounded local queues and explicit safe busy retries. Large files, CPU-heavy scripts, shared-sandbox contention, broader tenant mixes and long-duration operation need their own capacity targets and measurements.

Summary by cubic

Bounding ingress queues lets replicas absorb short request bursts while Azure Container Apps scales out, and records applied PostgreSQL migrations so unchanged replica startups skip table DDL.

  • New SQL_FS_MAX_INGRESS_QUEUED_REQUESTS and SQL_FS_MAX_INGRESS_TENANT_QUEUED_REQUESTS limit waiting requests per server and per tenant; zero disables waiting and defaults keep the current queue sizes.
  • Queued bodies are read within existing byte budgets, so client disconnects remove waiting requests; immediate queue-full refusals leave bodies unread and unknown outcomes are never replayed.
  • A failure in the queued-notification hook cancels that admission without leaking the queue entry.
  • Startup records migration filenames and SHA-256 checksums per tenant database, so unchanged replica boots skip DDL; modified applied files fail startup.
  • The runtime image drops development dependencies, and the ACA deployment example scales early with bounded five-second queues and references an immutable image digest.
  • Adds reproducible ACA load/readback tooling plus ingress queue, outcome, and metric test suites.

Migration

  • First adoption on an existing database applies the idempotent SQL files once; keep traffic idle during that boot because the files lock filesystem tables.

Summary by CodeRabbit

  • New Features
    • Queued HTTP and MCP requests now have configurable global and per-tenant limits. Waiting uploads are cancelled if the client disconnects, and queue activity and wait times are measurable.
    • Database migrations are recorded per tenant database. Unchanged migrations are skipped on startup, while changes to previously applied migrations are rejected.
  • Improvements
    • The Azure deployment example now uses more responsive scaling and readiness settings, a longer shutdown grace period, and disabled blob caching.
  • Documentation
    • Updated guidance covers queue limits, ingress metrics, migration tracking, autoscaling, and safe retry behavior.