feat(sandbox): acked delivery, exec deadlines and cancel-on-disconnect

BN
5d ago
+2148-286

Stacked on #619.

At-least-once work delivery. Poll returns at most PRELOOP_SANDBOX_POLL_BATCH items (default 64) and keeps them in flight keyed by work id until the agent reports; unacknowledged items are redelivered after PRELOOP_SANDBOX_DELIVERY_TIMEOUT_SECS (default 120) and when the provider re-registers. The agent dedupes by work id (running/done) and re-reports stored results instead of re-executing.

Reliable reports. The agent retries Report / ReportExec with jittered exponential backoff (cap 15 s) until success or fencing. ReportExec carries a per-exec sequence (generation-scoped); the service drops duplicates/out-of-order batches and closes the caller's stream on the exit event. A lost exit no longer hangs the caller.

Deadlines. timeout_secs = 0 gets PRELOOP_SANDBOX_EXEC_TIMEOUT_SECS (default 3600); drivers kill and report exit 124. Snapshot waits are bounded by the work timeout. Stale execs/pending are swept.

Caller disconnect. Dropping the exec stream queues CancelExec; the agent calls the new optional Driver::cancel_exec (SmolVM kills the exec child, exit 130) and the service drops its exec state.

Agent loop. Jittered poll backoff (0.5ā€“15 s); work runs under a semaphore (PRELOOP_SANDBOX_AGENT_CONCURRENCY, default 32).

Tests: batch cap, redelivery + re-register release, sequence ordering + close-on-exit, default deadline, disconnect cancel, lost ReportExec retried, redelivered create not re-executed, bounded snapshot wait; unit tests for the cancel race, jitter window and fencing.
Lockfile: one first-party edge (rand on preloop-sandbox; already locked).

Summary by cubic

Sandbox work delivery is now at-least-once and every exec is bounded by a deadline; disconnected callers cancel their exec instead of leaving it running.

Delivery and deadlines

  • Poll returns at most PRELOOP_SANDBOX_POLL_BATCH (default 64) items and keeps them in flight until acked by Report; unacked items are redelivered after PRELOOP_SANDBOX_DELIVERY_TIMEOUT_SECS (default 120 s) or when a provider re-registers.
  • Agents dedupe by work id and re-report stored results instead of re-executing; work runs under a semaphore (PRELOOP_SANDBOX_AGENT_CONCURRENCY, default 32).
  • Report and ReportExec are retried with jittered exponential backoff (capped 15 s); ReportExec batches carry per-exec sequences so retries are dropped.
  • timeout_secs = 0 means the service deadline (PRELOOP_SANDBOX_EXEC_TIMEOUT_SECS, default 3600 s). Drivers kill the command at it (exit 124), and overdue execs are swept to the same exit 124, never an error.
  • Provider and host-tool waits run through the shared bounded-deadline runners instead of unbounded child waits.

Cancel-on-disconnect

  • Dropping the exec stream queues CancelExec; the agent calls the new optional Driver::cancel_exec (SmolVM kills the exec child, exit 130) and the service drops the exec state.
  • SmolVM wraps streaming commands in a guest shell that records the PID and start time; on deadline or cancel a follow-up machine exec terminates the process group (SIGTERM then SIGKILL), guarding against PID reuse.
  • preloop sandbox exec installs a SIGINT listener before sending the request, so an inherited SIG_IGN process can still stop the pump loop and trigger the cancel path.