fix(sandbox): no swap on cloud boxes; memguard reaps early enough without it
Draft for the product owner's review. It replaces the closed draft #9843: it re-applies that change on current main and adds the memguard changes that the measurements below showed are needed.
What changed
superset-memguard: no swap. Removedgrow_swap/shrink_swap, the swap constants,box_mband the swap cap. At start it still turns off and removes any/var/swap/superset-*an earlier bundle left, then removes/var/swap.- memguard's loop, changed because the measurements showed removing swap alone is not safe (see "Why the loop changed"):
- It checks
MemAvailableevery 0.25 s (was 1 s), and nothing between checks forks: the pause isread -ton a pipe, notsleep. Ranking still runs once a second. - It reaps when
available − (fall since the last check)is under the floor, so a fast allocator is caught one check early. Before choosing the victim it does a fresh scan. - After a kill it waits for the victim to become a zombie (its memory is then free), at most 3 s. Before, it slept a fixed 3 s.
- It kills first and then logs (the log forks
date). The reap line gainsdrop=…MB; host-service's parser reads only the prefix, so the pressure banner is unchanged. - The floor is still 8% of the box.
- It checks
superset-boot: comment only. It still removes the old fixed/swapfile.boot.test.ts: the swap-cap test is replaced. New tests check that nothing turns swap on, that the old swapfiles are removed, the reap condition, that there is nosleepin the loop, and that it kills before it logs.- Docs:
cloud-sandbox-considerations.mdnow has "No swap (2026-11-17)" in place of "Swap is on demand", with the measurements and the open item for Vercel's OOM webhooks.cloud-sandbox-mismatches.md: the open item no longer says "swap included". - Step versions: none change. memguard and boot are rootfs files, and no step lists them as inputs. They reach a box when a bundle carrying them is released.
superset-bootreads only a comment change, so--skip-imageis fine for that file.
memguard's loop after the change
prev_avail = 0; check = 0
forever:
if check++ % 4 == 0: scan # rank every sandbox-user process (-900 / -300 / 500), once a second
read MemTotal, MemAvailable # builtin read of /proc/meminfo, no fork
floor = MemTotal * 8%
drop = max(prev_avail - avail, 0); prev_avail = avail
if avail - drop < floor:
scan # fresh RSS
victim = largest at adj >= 500, else >= 0, else >= -300 (min 32 MB)
if victim:
kill -9 victim; log "reap ..."
wait until victim is a zombie or gone (<= 3 s, 0.1 s steps)
continue # check again at once
log "pressure ... nothing to reap"
pause 0.25 s # read -t on a pipe, no fork
Why the loop changed (what the first runs showed)
- No memguard, no swap, steady allocator, 4 GB box: host-service went silent for 282 s, and the kernel's killer never fired. This is the stall memguard's header warns about.
- The draft #9843 as it was (no swap, 1 s check, fixed 3 s cooldown): it passed (a), (c) and, by luck, (d) once. It failed (b) at 8 GB: after the first reap, seven workers allocated about 2 GB during the 3 s cooldown, the box thrashed, memguard itself was starved, and host-service went silent for 175 s.
- The cooldown fix alone: (b) passed, but (d) at 4 GB failed. A 2.2 GB/s allocator filled the box between two 1 s checks, memguard was starved for 135 s, and the kernel livelocked without killing anything. That is why the check is now every 0.25 s with no forks, and why it projects one check ahead.
With and without swap: four cases at 4 GB and 8 GB
"yes (main)" is memguard from main with on-demand swap. "no (branch)" is this PR. Each (d) row has three or four runs, because (d) is a race.
| Box | Case | Swap | host-service max gap | host-service / pty-daemon / agent killed | Reaped by memguard (first, from start) | Kernel OOM kills | Peak swap | How the workload ended | Recovery |
|---|---|---|---|---|---|---|---|---|---|
| 4 GB | (a) one steady allocator | yes (main) | 272 ms | none | 1 (first +85.8s, 6356MB) | 0 | 5.4 GB | killed at 86 s | 207 ms |
| 4 GB | (a) one steady allocator | no (branch) | 267 ms | none | 1 (first +24.6s, 2366MB) | 0 | 0.0 GB | killed at 25 s | 184 ms |
| 4 GB | (b) parallel workers | yes (main) | 325 ms | none | 0 | 0 | 3.6 GB | finished at 37 s | 241 ms |
| 4 GB | (b) parallel workers | no (branch) | 255 ms | none | 3 (first +7.1s, 710MB) | 0 | 0.0 GB | 3 of workers reaped, rest finished at 36 s | 61 ms |
| 4 GB | (c) page cache + allocator | yes (main) | 378 ms | none | 1 (first +123.4s, 6196MB) | 0 | 6.0 GB | killed at 125 s | 203 ms |
| 4 GB | (c) page cache + allocator | no (branch) | 314 ms | none | 1 (first +32.1s, 2314MB) | 0 | 0.0 GB | killed at 33 s | 182 ms |
| 4 GB | (d) fast allocator | yes (main) | 651 ms, 6676 ms, 62317 ms | none | 0; 0; 0 | 1 (MainThread), 1 (MainThread), 1 (MainThread) | 0.0 GB, 0.0 GB, 0.0 GB | killed at 2 s; killed at 8 s; killed at 64 s | 77 ms, 110 ms, 294 ms |
| 4 GB | (d) fast allocator | no (branch) | 257 ms, 254 ms, 265 ms | none | 1 (first +1.2s, 2430MB); 1 (first +1.1s, 2502MB); 1 (first +1.5s, 3140MB) | 0, 0, 0 | 0.0 GB, 0.0 GB, 0.0 GB | killed at 1 s; killed at 1 s; killed at 2 s | 73 ms, 111 ms, 246 ms |
| 8 GB | (a) one steady allocator | yes (main) | 541 ms | none | 1 (first +170.1s, 8647MB) | 0 | 10.9 GB | killed at 171 s | 248 ms |
| 8 GB | (a) one steady allocator | no (branch) | 255 ms | none | 1 (first +54.9s, 5294MB) | 0 | 0.0 GB | killed at 55 s | 128 ms |
| 8 GB | (b) parallel workers | yes (main) | 664 ms | none | 0 | 0 | 8.2 GB | finished at 39 s | 18 ms |
| 8 GB | (b) parallel workers | no (branch) | 259 ms | none | 6 (first +6.0s, 611MB) | 0 | 0.0 GB | 6 of workers reaped, rest finished at 36 s | 217 ms |
| 8 GB | (c) page cache + allocator | yes (main) | 505 ms | none | 0 | 0 | 9.8 GB | timed out at 180 s (harness) | 211 ms |
| 8 GB | (c) page cache + allocator | no (branch) | 283 ms | none | 1 (first +62.9s, 4597MB) | 0 | 0.0 GB | killed at 65 s | 180 ms |
| 8 GB | (d) fast allocator | yes (main) | 2739 ms, 2591 ms, 93939 ms, 4996 ms | none | 0; 0; 0; 1 (first +22.6s, 9016MB) | 1 (MainThread), 1 (MainThread), 1 (MainThread), 0 | 0.0 GB, 0.0 GB, 0.0 GB, 12.0 GB | killed at 5 s; killed at 5 s; killed at 96 s; killed at 23 s | 82 ms, 102 ms, 164 ms, 157 ms |
| 8 GB | (d) fast allocator | no (branch) | 253 ms, 252 ms, 253 ms | none | 1 (first +1.9s, 4626MB); 1 (first +1.9s, 4590MB); 1 (first +1.7s, 4041MB) | 0, 0, 0 | 0.0 GB, 0.0 GB, 0.0 GB | killed at 2 s; killed at 2 s; killed at 2 s | 78 ms, 103 ms, 99 ms |
- host-service max gap: the longest time between two successful
/trpc/health.checkanswers, probed every 250 ms from outside the box's memory (a C prober,mlockall). - Killed: whether host-service, pty-daemon or the agent-class process (a 400 MB process named
claude, rank −300) died. None died in any run. A 300 MB "dev server" (rank 500) also survived every run, because the reaps always picked the larger allocator. - Recovery: from the last kill to four probes in a row under 100 ms.
- Peak swap: the VM's swap in use, which on main can include the test box's own idle pages.
- "MainThread": Node 24 names its main thread, so every node process has
comm=MainThread. See "What the user sees".
The workloads, run as the sandbox user: (a) one node process allocating 100 MB/s until it is killed. (b) 2×vCPU workers, each growing at 100 MB/s to a share of 1.5× the box, holding 20 s, then exiting, like a parallel build. (c) 0.75× the box of files written, then read in a loop together with every small file under /usr and /opt, while an allocator grows at 80 MB/s, like a large install. (d) one node process allocating 128 MB chunks as fast as it can, about 2.2 GB/s.
How this was measured, and what that means for trust
- Not in a container with a memory cgroup limit. That was the plan, and it cannot work in a cloud box: the sandbox's root cgroup is
domain threaded, so no child cgroup can enable the memory controller, anddocker run --memoryfails with "cannot enter cgroupv2 … in threaded mode". Anyone repeating this on a cloud workspace will hit the same wall. - What I did instead. The test ran on this workspace's VM, a real 8 vCPU / 16 GB Vercel box. Reserved hugepages (
vm.nr_hugepages) took RAM out of the VM so that what was left was 4 or 8 GB. The real image (superset-sandbox:local) ran in a privileged container pinned to 2 or 4 CPUs, booted with the realsuperset-boot: host-service, pty-daemon and the Xfce desktop all came up. Then the memguard under test replaced boot's copy. The only edit to memguard was to make it see the box size:MemTotalis set to 4 or 8 GB after the read, and on main the cgroup read for the swap cap is replaced by the same value. Swap on main was real (swapfiles on this VM's disk,vm.swappiness=10as memguard sets it). The kernel reclaim, page cache and OOM killer were the VM's own, which matches a real box better than a cgroup limit would. - Limits of the method. The VM's watermarks are those of 16 GB, not 4. This workspace's own processes (its dev stack, about 5 GB) shared the CPU and page cache. And this VM's disk stood in for the swap device of a smaller box.
- After every main run, the swapfiles were turned off, overwritten with zeros, then removed. So this workspace's snapshot did not keep them.
- Harness (not committed): it was kept in
/tmp/memtest; I can attach it on request.
What the user sees when something is reaped
- Desktop app, the open workspace: within about 5 s (host-service samples memguard.log and PSI every 5 s, and the API passes it to the realtime hub),
BoxOutOfMemoryBannershows: "Out of memory: the box stopped {process} ({n} GB) to keep running. {size} has {m} GB." Beside it is a "Resize to {next size}" button, which opens the machine-size menu with the suggested size, and a dismiss button. The sidebar shows a pressure dot on that workspace in every window. - The terminal that ran the process shows
Killedfrom the shell, with exit 137. An agent sees exit 137 from its tool. - Does it tell them the remedy? Partly. The button offers a larger size, but the text never says that a larger size is the fix.
- Follow-ups (not built here):
- The process name is useless for node. Node 24 sets
commtoMainThread, so a reaped Vite or Next.js dev server shows as "the box stopped MainThread (2.4 GB)". Next.js shows asnext-server (v1(cut to 15 characters). memguard could log the script from argv (it already reads it for ranking), and the banner could show that. - On the largest size there is no remedy.
nextMachineSizereturns the current size, so no button appears and the text gives no next step, such as "run fewer jobs in parallel". - The banner is desktop only; web and mobile show nothing.
- The process name is useless for node. Node 24 sets
- Vercel OOM webhooks: not built. Per the decision, the note is in
cloud-sandbox-considerations.md. The API would report them to the realtime hub beside host-service's report (runtime/sandbox-box-pressure). They would also cover a box whose host-service is itself stalled, which host-service's own report cannot.
Product implications
- What gets killed sooner. A job bigger than the box. Before, it swapped and often finished slowly: on main, (b) finished at both sizes, and (c) at 8 GB was still running at 180 s with 9.8 GB of swap. Now the largest rank-500 process is killed once available memory falls to about 8% of the box, plus one check's fall. Steady jobs died at 25 s (4 GB) and 55 s (8 GB), where on main they ran 86 s and 170 s before the same kill. Parallel builds lose their largest workers (3 of 4 at 4 GB, 6 of 8 at 8 GB) instead of finishing.
- On which sizes. All of them, but it matters most on the 2 vCPU / 4 GB size: a monorepo typecheck (about 8 GB, from #9843's measurements) will be killed there. It needed swap to finish. 8 GB boxes lose jobs above about 7.4 GB. 16 GB boxes are rarely affected.
- What gets better. The fast-allocator case no longer gives the kernel's killer a 6–94 s stall first. With swap and a 1 s check, that happened in 6 of 7 runs. And no box writes swap to its disk.
- Agents and the desktop (rank −300) and host-service / pty-daemon (−900) were never touched in any run.
Snapshot saving
Swap was capped at 1.5× box RAM: 6 GB on a 4 GB box, 12 GB on 8 GB, 16 GB on 16 GB. A snapshot is a full block copy with no discard, so a workspace that ever came under pressure carried up to that much in every later snapshot. In these runs main wrote 3.6–6.0 GB of swap at 4 GB and 8.2–12.0 GB at 8 GB. After this change, any workspace that hits pressure from now on saves that, up to 6–16 GB per workspace. Workspaces that never grew swap save nothing. Snapshots that already hold swap keep it: removing the file does not zero its blocks.
Decisions that are yours
- Ship no swap with this loop? The trade is above: bigger-than-box jobs die early instead of finishing slowly, in exchange for no snapshot growth and no long stalls from fast allocators.
- The floor stays 8%. The early reaping comes from the rate projection, not a higher floor. A higher floor would kill sooner on every box; a lower one leaves less margin. 8% passed all cases.
- The banner follow-ups above: the process name, and the text when there is no larger size. Should they be scheduled?
- A one-week telemetry check before merging? The question is how many boxes ever reaped or grew swap, sent through the existing pressure report. #9843 suggested this; it is still unanswered.
Checks
Ran:
bun testinpackages/sandbox: 66 pass, 0 fail.bash -n superset-memguard: passed.bun run lint:fix: passed, no fixes.- Typecheck, scoped to
packages/sandbox(bun run typecheckin the package): passed. - The pressure matrix above: 4 cases × 2 sizes × with and without swap, (d) 3–4 times each; run with this PR's memguard code (the only later edit is one comment line).
- In the container, memguard started, logged
startand ran; the fork-free pause measured 251 ms for 0.25 s. bun run --cwd packages/sandbox image --localon this branch: builtsuperset-sandbox:local(4.89 GB); the image's memguard is this PR's.src/runner-check.ts: all expectations met (30 ok).src/boot-twice.ts: 3 expectations failed, the rest pass. Boot, host-service (ready in 1193 ms, then 899 ms), the desktop,memguard.startwithmemguard.pid, the second-boot skips, and the drive cases all pass. The failures arefirst boot: checkout done,start hook: the runner asked host-service once the push landedandsecond boot: checkout marker respected. All three come fromcheckout.clone.failed: inside the container,git ls-remote https://github.com/paradevai/paradev.gitfails with "server certificate verification failed", because this workspace's egress needs a CA the image does not have. The same command on the host succeeds. These are the same three failures #9843 reported, and memguard is not involved.
Not run:
- Whole-monorepo
bun run typecheck. It needs about 8 GB, and this workspace's own memguard is on the old bundle, so it would grow swap on this disk. The change touches onlypackages/sandboxand docs. - Any real Vercel sandbox (no credentials here), and snapshot size before and after.
- boot-twice on
mainin this workspace, for a side-by-side comparison of the three clone failures above (their cause is shown directly instead).