Run workers by default with one 64 MiB exec output cap

Merged
HA
7d ago
+228-10733

Closes #231.

What changes for deployments

  • Workers run by default. Set SQL_FS_WORKERS_ENABLED=false to run scripts on the event loop instead. Any value other than true or false stops startup.
  • One output cap for both modes. SQL_FS_WORKER_OUTPUT_BYTES (64 MiB) now also caps stdout plus stderr when workers are off. Before, inline mode passed no limit and got the patched just-bash default of 256 MiB, not the 10 MiB the issue assumed.
  • The message budget follows the output cap. SQL_FS_WORKER_MESSAGE_BYTES defaults to the output cap plus 1 MiB, instead of a fixed 65 MiB. If you raise only the output cap, normal replies no longer fail with EEXEC_OUTCOME_UNKNOWN. An explicit value still wins.
  • Upgrades flip the mode. A deployment that never set the flag now gets worker mode and ingress admission. Check the worker rollout prerequisites in docs/CONFIGURATION.md before you roll this out alongside older replicas.

Why 64 MiB and not more

Measured on a 2 GiB container with 2 CPUs, no swap and the default 2 workers. Two tenants each ran one exec-sync at once; the table shows peak cgroup memory.

Output capprintf zeros\x01 bytesNewlines
64 MiB747 MiB1078 MiB1446 MiB
96 MiB–1528 MiBOOM-killed
128 MiB1326 MiB1693 MiBOOM-killed

Output that is mostly newlines costs about 11 times its size per exec. An OOM kill takes down the whole replica. So 64 MiB is the largest cap that leaves headroom on 2 GiB. Run more replicas rather than raising the cap. The new Sizing section in docs/CONFIGURATION.md has the full table, the upload and 100-agent measurements, and the metrics to autoscale on. #267 tracks a per-replica output-byte budget, which would let a box run more workers without assuming every exec hits the worst case.

Commits

  1. feat(api): server.ts reads the flag once and passes it to SessionManager and to ingress admission. SessionManager no longer reads the environment and defaults to inline mode, so unit tests with in-memory filesystems keep working. src/api/lib/env.ts now holds the strict integer and boolean helpers and loadExecOutputBytes. SQL_FS_WORKER_LARGE_FILES uses the same boolean helper.
  2. docs: README, .env.example, docs/CONFIGURATION.md, docs/DEVELOPER.md and the changeset.
  3. chore: removes security/audits and thoughts/shared, and fixes the references to them in SECURITY.md, AGENTS.md, docs/DEVELOPER.md and docs/READINESS.md.

Tests

  • scripts/verify/output-cap.mjs runs in verify:local. It starts a real server in each mode with a 1 MiB cap. Output under the cap returns in full. Output over it fails identically in both modes, with exit code 126 and the total output size exceeded error.
  • server.workers-enabled.test.ts checks /metrics with the flag unset, true and false, and checks that an invalid value fails startup. env.test.ts covers the boolean helper's invalid inputs.
  • worker-resource-policy.test.ts checks that the message budget follows the output cap and that an explicit value overrides it.

Each new test failed with its change reverted.

Verification

  • pnpm check passed: 2327 and 154 tests.
  • pnpm verify:architecture passed: contracts 94, output-cap 19, integration 265 (none skipped), SDK clients 102, worker failures 110, SSE finalization 109, input budgets 283, tenants 27.

The sizing numbers come from one-off probes, not a script in the repo. The case of slow clients holding large responses is not measured; #267 covers it.

Summary by cubic

Workers now run by default, capped at a single 64 MiB exec output limit shared by worker and inline modes.

  • Set SQL_FS_WORKERS_ENABLED=false for inline mode; any value other than true or false stops startup.
  • Both modes cap output at SQL_FS_WORKER_OUTPUT_BYTES (64 MiB), down from the 256 MiB inline default.
  • SQL_FS_WORKER_MESSAGE_BYTES now defaults to the output cap plus 1 MiB, with a 65 MiB floor so large writeFile commands still fit the shared budget when the cap is lowered.
  • A deployment that never set the flag now gets worker mode and ingress admission; review the staged rollout prerequisites in docs/CONFIGURATION.md before mixing this version with older replicas.
  • Removes the tracked security/audits snapshot and thoughts/shared historical records.

Why 64 MiB

  • Measured on a 2 GiB container, newline-heavy output costs roughly 11× its size per exec; a 96 MiB cap OOM-killed the container.
  • The full sizing table and autoscaling metrics are in the new Sizing section of docs/CONFIGURATION.md.

Summary by CodeRabbit

  • New Features
    • Worker threads are now enabled by default. Set SQL_FS_WORKERS_ENABLED=false to run scripts on the HTTP event loop; invalid values prevent startup.
    • Script output is capped at 64 MiB by default in both modes. The cap can be configured, and the default worker message limit adjusts to match it.
  • Documentation
    • Updated configuration and sizing guidance, including upgrade considerations and deployment requirements.