fix(sandbox): no swap on cloud boxes; memguard reaps early enough without it

Merged
SA
21h ago
+86-87

Draft for the product owner's review. It replaces the closed draft #9843: it re-applies that change on current main and adds the memguard changes that the measurements below showed are needed.

What changed

  • superset-memguard: no swap. Removed grow_swap/shrink_swap, the swap constants, box_mb and the swap cap. At start it still turns off and removes any /var/swap/superset-* an earlier bundle left, then removes /var/swap.
  • memguard's loop, changed because the measurements showed removing swap alone is not safe (see "Why the loop changed"):
    • It checks MemAvailable every 0.25 s (was 1 s), and nothing between checks forks: the pause is read -t on a pipe, not sleep. Ranking still runs once a second.
    • It reaps when available − (fall since the last check) is under the floor, so a fast allocator is caught one check early. Before choosing the victim it does a fresh scan.
    • After a kill it waits for the victim to become a zombie (its memory is then free), at most 3 s. Before, it slept a fixed 3 s.
    • It kills first and then logs (the log forks date). The reap line gains drop=…MB; host-service's parser reads only the prefix, so the pressure banner is unchanged.
    • The floor is still 8% of the box.
  • superset-boot: comment only. It still removes the old fixed /swapfile.
  • boot.test.ts: the swap-cap test is replaced. New tests check that nothing turns swap on, that the old swapfiles are removed, the reap condition, that there is no sleep in the loop, and that it kills before it logs.
  • Docs: cloud-sandbox-considerations.md now has "No swap (2026-11-17)" in place of "Swap is on demand", with the measurements and the open item for Vercel's OOM webhooks. cloud-sandbox-mismatches.md: the open item no longer says "swap included".
  • Step versions: none change. memguard and boot are rootfs files, and no step lists them as inputs. They reach a box when a bundle carrying them is released. superset-boot reads only a comment change, so --skip-image is fine for that file.

memguard's loop after the change

prev_avail = 0; check = 0
forever:
    if check++ % 4 == 0: scan          # rank every sandbox-user process (-900 / -300 / 500), once a second
    read MemTotal, MemAvailable         # builtin read of /proc/meminfo, no fork
    floor = MemTotal * 8%
    drop  = max(prev_avail - avail, 0); prev_avail = avail
    if avail - drop < floor:
        scan                            # fresh RSS
        victim = largest at adj >= 500, else >= 0, else >= -300 (min 32 MB)
        if victim:
            kill -9 victim; log "reap ..."
            wait until victim is a zombie or gone (<= 3 s, 0.1 s steps)
            continue                    # check again at once
        log "pressure ... nothing to reap"
    pause 0.25 s                        # read -t on a pipe, no fork

Why the loop changed (what the first runs showed)

  1. No memguard, no swap, steady allocator, 4 GB box: host-service went silent for 282 s, and the kernel's killer never fired. This is the stall memguard's header warns about.
  2. The draft #9843 as it was (no swap, 1 s check, fixed 3 s cooldown): it passed (a), (c) and, by luck, (d) once. It failed (b) at 8 GB: after the first reap, seven workers allocated about 2 GB during the 3 s cooldown, the box thrashed, memguard itself was starved, and host-service went silent for 175 s.
  3. The cooldown fix alone: (b) passed, but (d) at 4 GB failed. A 2.2 GB/s allocator filled the box between two 1 s checks, memguard was starved for 135 s, and the kernel livelocked without killing anything. That is why the check is now every 0.25 s with no forks, and why it projects one check ahead.

With and without swap: four cases at 4 GB and 8 GB

"yes (main)" is memguard from main with on-demand swap. "no (branch)" is this PR. Each (d) row has three or four runs, because (d) is a race.

BoxCaseSwaphost-service max gaphost-service / pty-daemon / agent killedReaped by memguard (first, from start)Kernel OOM killsPeak swapHow the workload endedRecovery
4 GB(a) one steady allocatoryes (main)272 msnone1 (first +85.8s, 6356MB)05.4 GBkilled at 86 s207 ms
4 GB(a) one steady allocatorno (branch)267 msnone1 (first +24.6s, 2366MB)00.0 GBkilled at 25 s184 ms
4 GB(b) parallel workersyes (main)325 msnone003.6 GBfinished at 37 s241 ms
4 GB(b) parallel workersno (branch)255 msnone3 (first +7.1s, 710MB)00.0 GB3 of workers reaped, rest finished at 36 s61 ms
4 GB(c) page cache + allocatoryes (main)378 msnone1 (first +123.4s, 6196MB)06.0 GBkilled at 125 s203 ms
4 GB(c) page cache + allocatorno (branch)314 msnone1 (first +32.1s, 2314MB)00.0 GBkilled at 33 s182 ms
4 GB(d) fast allocatoryes (main)651 ms, 6676 ms, 62317 msnone0; 0; 01 (MainThread), 1 (MainThread), 1 (MainThread)0.0 GB, 0.0 GB, 0.0 GBkilled at 2 s; killed at 8 s; killed at 64 s77 ms, 110 ms, 294 ms
4 GB(d) fast allocatorno (branch)257 ms, 254 ms, 265 msnone1 (first +1.2s, 2430MB); 1 (first +1.1s, 2502MB); 1 (first +1.5s, 3140MB)0, 0, 00.0 GB, 0.0 GB, 0.0 GBkilled at 1 s; killed at 1 s; killed at 2 s73 ms, 111 ms, 246 ms
8 GB(a) one steady allocatoryes (main)541 msnone1 (first +170.1s, 8647MB)010.9 GBkilled at 171 s248 ms
8 GB(a) one steady allocatorno (branch)255 msnone1 (first +54.9s, 5294MB)00.0 GBkilled at 55 s128 ms
8 GB(b) parallel workersyes (main)664 msnone008.2 GBfinished at 39 s18 ms
8 GB(b) parallel workersno (branch)259 msnone6 (first +6.0s, 611MB)00.0 GB6 of workers reaped, rest finished at 36 s217 ms
8 GB(c) page cache + allocatoryes (main)505 msnone009.8 GBtimed out at 180 s (harness)211 ms
8 GB(c) page cache + allocatorno (branch)283 msnone1 (first +62.9s, 4597MB)00.0 GBkilled at 65 s180 ms
8 GB(d) fast allocatoryes (main)2739 ms, 2591 ms, 93939 ms, 4996 msnone0; 0; 0; 1 (first +22.6s, 9016MB)1 (MainThread), 1 (MainThread), 1 (MainThread), 00.0 GB, 0.0 GB, 0.0 GB, 12.0 GBkilled at 5 s; killed at 5 s; killed at 96 s; killed at 23 s82 ms, 102 ms, 164 ms, 157 ms
8 GB(d) fast allocatorno (branch)253 ms, 252 ms, 253 msnone1 (first +1.9s, 4626MB); 1 (first +1.9s, 4590MB); 1 (first +1.7s, 4041MB)0, 0, 00.0 GB, 0.0 GB, 0.0 GBkilled at 2 s; killed at 2 s; killed at 2 s78 ms, 103 ms, 99 ms
  • host-service max gap: the longest time between two successful /trpc/health.check answers, probed every 250 ms from outside the box's memory (a C prober, mlockall).
  • Killed: whether host-service, pty-daemon or the agent-class process (a 400 MB process named claude, rank −300) died. None died in any run. A 300 MB "dev server" (rank 500) also survived every run, because the reaps always picked the larger allocator.
  • Recovery: from the last kill to four probes in a row under 100 ms.
  • Peak swap: the VM's swap in use, which on main can include the test box's own idle pages.
  • "MainThread": Node 24 names its main thread, so every node process has comm = MainThread. See "What the user sees".

The workloads, run as the sandbox user: (a) one node process allocating 100 MB/s until it is killed. (b) 2×vCPU workers, each growing at 100 MB/s to a share of 1.5× the box, holding 20 s, then exiting, like a parallel build. (c) 0.75× the box of files written, then read in a loop together with every small file under /usr and /opt, while an allocator grows at 80 MB/s, like a large install. (d) one node process allocating 128 MB chunks as fast as it can, about 2.2 GB/s.

How this was measured, and what that means for trust

  • Not in a container with a memory cgroup limit. That was the plan, and it cannot work in a cloud box: the sandbox's root cgroup is domain threaded, so no child cgroup can enable the memory controller, and docker run --memory fails with "cannot enter cgroupv2 … in threaded mode". Anyone repeating this on a cloud workspace will hit the same wall.
  • What I did instead. The test ran on this workspace's VM, a real 8 vCPU / 16 GB Vercel box. Reserved hugepages (vm.nr_hugepages) took RAM out of the VM so that what was left was 4 or 8 GB. The real image (superset-sandbox:local) ran in a privileged container pinned to 2 or 4 CPUs, booted with the real superset-boot: host-service, pty-daemon and the Xfce desktop all came up. Then the memguard under test replaced boot's copy. The only edit to memguard was to make it see the box size: MemTotal is set to 4 or 8 GB after the read, and on main the cgroup read for the swap cap is replaced by the same value. Swap on main was real (swapfiles on this VM's disk, vm.swappiness=10 as memguard sets it). The kernel reclaim, page cache and OOM killer were the VM's own, which matches a real box better than a cgroup limit would.
  • Limits of the method. The VM's watermarks are those of 16 GB, not 4. This workspace's own processes (its dev stack, about 5 GB) shared the CPU and page cache. And this VM's disk stood in for the swap device of a smaller box.
  • After every main run, the swapfiles were turned off, overwritten with zeros, then removed. So this workspace's snapshot did not keep them.
  • Harness (not committed): it was kept in /tmp/memtest; I can attach it on request.

What the user sees when something is reaped

  • Desktop app, the open workspace: within about 5 s (host-service samples memguard.log and PSI every 5 s, and the API passes it to the realtime hub), BoxOutOfMemoryBanner shows: "Out of memory: the box stopped {process} ({n} GB) to keep running. {size} has {m} GB." Beside it is a "Resize to {next size}" button, which opens the machine-size menu with the suggested size, and a dismiss button. The sidebar shows a pressure dot on that workspace in every window.
  • The terminal that ran the process shows Killed from the shell, with exit 137. An agent sees exit 137 from its tool.
  • Does it tell them the remedy? Partly. The button offers a larger size, but the text never says that a larger size is the fix.
  • Follow-ups (not built here):
    1. The process name is useless for node. Node 24 sets comm to MainThread, so a reaped Vite or Next.js dev server shows as "the box stopped MainThread (2.4 GB)". Next.js shows as next-server (v1 (cut to 15 characters). memguard could log the script from argv (it already reads it for ranking), and the banner could show that.
    2. On the largest size there is no remedy. nextMachineSize returns the current size, so no button appears and the text gives no next step, such as "run fewer jobs in parallel".
    3. The banner is desktop only; web and mobile show nothing.
  • Vercel OOM webhooks: not built. Per the decision, the note is in cloud-sandbox-considerations.md. The API would report them to the realtime hub beside host-service's report (runtime/sandbox-box-pressure). They would also cover a box whose host-service is itself stalled, which host-service's own report cannot.

Product implications

  • What gets killed sooner. A job bigger than the box. Before, it swapped and often finished slowly: on main, (b) finished at both sizes, and (c) at 8 GB was still running at 180 s with 9.8 GB of swap. Now the largest rank-500 process is killed once available memory falls to about 8% of the box, plus one check's fall. Steady jobs died at 25 s (4 GB) and 55 s (8 GB), where on main they ran 86 s and 170 s before the same kill. Parallel builds lose their largest workers (3 of 4 at 4 GB, 6 of 8 at 8 GB) instead of finishing.
  • On which sizes. All of them, but it matters most on the 2 vCPU / 4 GB size: a monorepo typecheck (about 8 GB, from #9843's measurements) will be killed there. It needed swap to finish. 8 GB boxes lose jobs above about 7.4 GB. 16 GB boxes are rarely affected.
  • What gets better. The fast-allocator case no longer gives the kernel's killer a 6–94 s stall first. With swap and a 1 s check, that happened in 6 of 7 runs. And no box writes swap to its disk.
  • Agents and the desktop (rank −300) and host-service / pty-daemon (−900) were never touched in any run.

Snapshot saving

Swap was capped at 1.5× box RAM: 6 GB on a 4 GB box, 12 GB on 8 GB, 16 GB on 16 GB. A snapshot is a full block copy with no discard, so a workspace that ever came under pressure carried up to that much in every later snapshot. In these runs main wrote 3.6–6.0 GB of swap at 4 GB and 8.2–12.0 GB at 8 GB. After this change, any workspace that hits pressure from now on saves that, up to 6–16 GB per workspace. Workspaces that never grew swap save nothing. Snapshots that already hold swap keep it: removing the file does not zero its blocks.

Decisions that are yours

  1. Ship no swap with this loop? The trade is above: bigger-than-box jobs die early instead of finishing slowly, in exchange for no snapshot growth and no long stalls from fast allocators.
  2. The floor stays 8%. The early reaping comes from the rate projection, not a higher floor. A higher floor would kill sooner on every box; a lower one leaves less margin. 8% passed all cases.
  3. The banner follow-ups above: the process name, and the text when there is no larger size. Should they be scheduled?
  4. A one-week telemetry check before merging? The question is how many boxes ever reaped or grew swap, sent through the existing pressure report. #9843 suggested this; it is still unanswered.

Checks

Ran:

  • bun test in packages/sandbox: 66 pass, 0 fail.
  • bash -n superset-memguard: passed.
  • bun run lint:fix: passed, no fixes.
  • Typecheck, scoped to packages/sandbox (bun run typecheck in the package): passed.
  • The pressure matrix above: 4 cases × 2 sizes × with and without swap, (d) 3–4 times each; run with this PR's memguard code (the only later edit is one comment line).
  • In the container, memguard started, logged start and ran; the fork-free pause measured 251 ms for 0.25 s.
  • bun run --cwd packages/sandbox image --local on this branch: built superset-sandbox:local (4.89 GB); the image's memguard is this PR's.
  • src/runner-check.ts: all expectations met (30 ok).
  • src/boot-twice.ts: 3 expectations failed, the rest pass. Boot, host-service (ready in 1193 ms, then 899 ms), the desktop, memguard.start with memguard.pid, the second-boot skips, and the drive cases all pass. The failures are first boot: checkout done, start hook: the runner asked host-service once the push landed and second boot: checkout marker respected. All three come from checkout.clone.failed: inside the container, git ls-remote https://github.com/paradevai/paradev.git fails with "server certificate verification failed", because this workspace's egress needs a CA the image does not have. The same command on the host succeeds. These are the same three failures #9843 reported, and memguard is not involved.

Not run:

  • Whole-monorepo bun run typecheck. It needs about 8 GB, and this workspace's own memguard is on the old bundle, so it would grow swap on this disk. The change touches only packages/sandbox and docs.
  • Any real Vercel sandbox (no credentials here), and snapshot size before and after.
  • boot-twice on main in this workspace, for a side-by-side comparison of the three clone failures above (their cause is shown directly instead).