improvement(execution): run sandbox workers on a dedicated worker host
Summary
- The isolated-vm pool can open each worker as a WebSocket connection to a dedicated worker host instead of a local child process. The host (
lib/execution/isolated-vm-host.cjs, shipped in the app image and run withnode) spawns the unchangedisolated-vm-worker.cjsper connection and relays its IPC messages verbatim. - Remote mode turns on only where
IVM_REMOTE_WORKER_URL+IVM_REMOTE_WORKER_TOKENare set; with the URL unset nothing changes and no flag lookup happens. The AppConfig-only kill switchisolated-vm-local-workers(no env fallback) forces local workers for rollback, and an unreachable AppConfig keeps the host in use. - With a host configured, an unreachable host fails executions as a system error; it never silently falls back to a local worker. Flipping the kill switch retires workers on the old target after their in-flight executions finish. If a target change removes a selected worker before dispatch, the execution is re-queued.
fetch()and brokers stay in the app, so the host only relays frames. The client sends the settings the app owns in its first frame: limits (IVM_MAX_STDOUT_CHARS,IVM_MAX_FETCH_OPTIONS_JSON_CHARS),TZ,LANGandLC_ALL. Both modes therefore enforce the same limits and give the same Date/Intl results. The host validates each value's shape.- Client uses the standard global
WebSocket(Bun and Node 22+); the host usesws, now declared by apps/sim and copied into the image. The connection authenticates with a token in its first frame. The host sends a heartbeat frame so a client can detect a dead host, and announces draining on SIGTERM. - ffmpeg/ffprobe now run with a minimal environment (PATH, HOME, fontconfig vars) and
-protocol_whitelist file,pipebefore every input. Inputs are always temp files we write; the lavfi and concat inputs still work.
Type of Change
- Improvement
Testing
-
isolated-vm.test.ts: two new tests. A configured host never falls back to a local worker when unreachable (fails with a fallback added). The kill switch forces local workers (fails if the flag is ignored). -
ffmpeg.test.ts: new test that ffmpeg/ffprobe get no app env (fails without theenvoption); argv expectations updated -
bun run test:isolated-vm-remote:e2e, run manually (not wired to CI): real host processes over real WebSockets under Bun. 17/17 checks pass: execution, worker parented by the host, limits come from the app, no host env in the worker, brokered fetch, a 5 MB result, timeout, cancel, auth rejection, an unparseable request target, protocol violations, capacity, crash → system error, drain, and fail-closed once the host is gone. Writes a JSON report toIVM_REMOTE_E2E_REPORT_PATH. -
Parity corpus run in both modes, all 12 results identical (result, stdout, error name/message/line, termination): objects,
undefined, console output, a thrown error, a syntax error, params, Date/Intl with TZ set only on the app, a timeout, brokered fetch, a 300k-char stdout truncation, a 2 MB result, async/await -
Latency over loopback, two runs per mode (500 sequential executions with 20 KB params). The p50 deltas below are remote minus local, from run 1 and run 2.
local remote cold start 66 ms 53–56 ms p50 1.28–1.38 ms 1.45–1.69 ms (+0.07 / +0.41 ms) p99 2.5–2.6 ms 2.6–3.6 ms 500 concurrent 458–483 ms 512–513 ms 40 CPU-bound concurrent 117–130 ms 119–123 ms -
Checked locally that the whitelist keeps the lavfi silent-audio input and the concat-list input working, and rejects nested http inputs
-
biome, comment-hygiene, utils, source-conventions, test-patterns and file-names pass locally; type-check and the full suite run in CI
Checklist
- Code follows project style guidelines
- Self-reviewed my changes
- Tests added/updated and passing (new tests pass the
test-auditauthoring gate) - No new warnings introduced
- I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)