fix(sandbox): drop on-demand swap from memguard (draft: trade-off for review)

Draft
SA
2d ago
+38-79

Draft: a human decides whether this ships. It trades speed on undersized boxes for smaller snapshots.

What changed

  • superset-memguard: removed grow_swap / shrink_swap, the swap constants (SWAP_DIR, SWAP_STEP_MB, SWAP_IDLE_S), box_mb and the swap cap, and the SwapTotal / SwapFree reads.
  • The reap condition was MemAvailable < floor && SwapFree < SwapTotal/10 + 1. It is now MemAvailable < floor (8% of the box). The old swap check was there to wait for swap to fill. With no swap, MemAvailable alone is the right signal. Ranking, victim choice (500 → 0 → −300), the cooldown and the log lines did not change.
  • Kept: at startup memguard still runs swapoff on any /var/swap/superset-* an older bundle left and removes it. superset-boot still removes the old /swapfile.
  • boot.test.ts: the swap-cap test is replaced with two tests. Nothing in boot or memguard turns swap on, and the stale-file cleanup and the boot /swapfile removal are still there. memguard reaps on available memory alone.
  • docs/cloud-sandbox-considerations.md: the "Swap is on demand" entry is replaced by "No swap", which keeps the history and records the open telemetry question.

The trade

  • Before: an undersized box gets slow but finishes the job, and its snapshots grow by up to 16 GB, permanently. A snapshot is a full block copy of the disk with no discard, so swap written once stays in every later snapshot of the workspace. A box measured today held 8 GB of it, and fleet snapshots of 14-27 GB exist.
  • After: on an undersized box, a dev server or build is killed earlier, with a reap pid=… rss=… available=… floor=… line in memguard.log, and the app's pressure banner shows it. The remedy is a larger machine size.
  • Measured while writing this PR, on this workspace (8 GB box, old bundle): a full bun run typecheck of the monorepo made memguard grow swap 0 → 12 GB (its cap) in about 60 s. free showed 4.3 GB of swap in use afterwards, so at least that much is now written to this workspace's disk and every later snapshot will copy it. Under this change, the same typecheck on the same box would probably have had its largest tsc reaped. Earlier measurements (2026-10-22, 2/4 box): the apps/desktop typecheck needed about 8 GB and finished in 86 s only because of swap. Expect it to be reaped on a 2/4 box now.
  • Expected saving: up to the swap cap (6-16 GB) per snapshot for any workspace that ever came under pressure. Workspaces that never grew swap save nothing. This change does not shrink snapshots that already hold swap: removing the file frees the space for reuse but does not zero it.

Alternatives not taken

  • zram: the sandbox kernel has no zram module.
  • Swap on a Vercel Drive, once workspaces have one: drives are not part of the snapshot, so this would keep swap without the snapshot cost. It is not available yet. This is the likely long-term answer.
  • Zero-filling swap at stop: stop gives SIGTERM and at least 10 s. Zeroing up to 16 GB does not reliably fit, and it needs a lot of new logic in a fail-open script.

What would settle it

How many boxes ever grow swap, and how often swap is what saved a job. memguard.log is box-local today, so we do not know. Shipping the swap.grow count (or the peak swap use) through host-service's existing pressure report (runtime/sandbox-box-pressure) to telemetry for a week, before or instead of merging, would answer it.

Checks

Ran:

  • bun run lint:fix: passed (it reformatted the new test).
  • bun run typecheck (whole monorepo, 42 tasks): passed.
  • bun test in packages/sandbox: 25 pass, 0 fail.
  • bash -n superset-memguard: passed.
  • bun run --cwd packages/sandbox image --local (Docker started on this box): built superset-sandbox:local.
  • src/runner-check.ts: all expectations met.
  • src/boot-twice.ts: memguard starts (memguard.start, memguard.pid present), and the boot, host-service and second-boot skip expectations pass. 3 expectations failed: first boot: checkout done, start hook: the runner asked host-service once the push landed, second boot: checkout marker respected. They come from checkout.clone.failed: the container cannot clone from GitHub here (git ls-remote in the image fails with "server certificate verification failed"; this workspace's egress needs a CA the container does not have). memguard is not involved, but I did not run boot-twice on main in this environment to confirm the same failures happen there.
  • In the built image, as root: I created /var/swap/superset-0 and ran memguard for 3 s. It logged start and removed the file.

Not run:

  • Any real Vercel sandbox (no credentials in this workspace). The reap-instead-of-swap behaviour under real memory pressure has not been measured. The "after" behaviour above is expected, not observed.
  • Snapshot size before and after.

Summary by cubic

Drops on-demand swap from memguard so snapshots stop permanently containing swap data. memguard now reaps when available memory drops below the 8% floor instead of growing swapfiles first, because snapshots are full block copies with no discard: swap written once (up to 16 GB per box) was being copied into every later snapshot of the workspace.

The trade-off

  • Before: an undersized box finished slowly but its snapshots grew by up to 16 GB permanently (a measured 8 GB on one box; fleet snapshots of 14–27 GB exist).
  • After: on an undersized box the largest process is reaped earlier, with a reap line in memguard.log; the remedy is a larger machine size.
  • Startup cleanup is kept: memguard still removes /var/swap/superset-* from older bundles, and boot still removes the old /swapfile.

Alternatives not taken

  • zram: the sandbox kernel has no zram module.
  • Swap on a Vercel Drive once workspaces have one: drives are not part of the snapshot, so this would keep swap without the snapshot cost — the likely long-term answer.
  • Zero-filling swap at stop: stop gives SIGTERM and at least 10 s, and zeroing up to 16 GB does not reliably fit.

How often swap actually saved a job is unknown because memguard.log is box-local; reporting swap.grow counts through host-service's pressure report for a week would settle it.