feat(cloud-workspace): own workspace retention with a daily reaper (draft)

Draft
SA
4m ago
+1952-25

Draft. Plan: plans/done/20261118-1500-cloud-workspace-retention-reaper.md.

What it does

  • New workspace boxes are created with snapshotExpiration: 0, as goldens already were. Vercel no longer deletes a workspace's snapshot; we decide when it goes.
  • A job that runs every 6 hours, reap-retention, does the deleting:
    • A workspace that is asleep and unused for 30 days becomes failed / snapshot_expired. That is the same state a 410 gives today, so customers see the same retention and the same failed screen.
    • First retireAndTearDown(row, condition) gives the row a new box name, so nothing can wake the box being deleted, and queues the old box's retried teardown. It has #9847's name, signature and module (@superset/trpc/cloud-workspace-teardown), box-only, so #9847's file replaces it as is.
    • Then the row fails. A wake between the two writes finds no box and fails the row itself.
    • Every box, with a drive or without, goes through the retried teardown job (deleteSandboxAndDriveLater). Main deleted boxes inline, without retry.
  • In production, the same job also deletes an archived box still there a day after its 7-day grace. Vercel's 30-day expiry used to clean up after a lost archive delete; with snapshotExpiration: 0 that backstop is gone. It goes through the archive's own delete stage: reapArchivedCloudWorkspace({stage: "delete", archivedAt: deleted_at}), or reapUndatedArchivedCloudWorkspace (archived, deleted_at null, updated_at older than the cutoff) for the 71 rows with no archive date. At most 20 a run, reported as found / alreadyDeleted / deleted / skipped / deferred / failed.
  • In production, it also takes the box from a row failed 30 days ago whose box is still listed, and from a row stuck in provisioning for a day (every delivery of its job died). It goes through retireAndTearDown with the status and age it read as the condition. A stranded provisioning row then fails as create_failed. At most 50 a run, reported as found / retired / skippedChanged / deferred / failed. This is the box half of "an expired workspace's drive is eventually deleted" (#9914's model); the drive half needs #9847's column (see below).
  • The same job queues teardown for orphan boxes in production. An orphan is a tagged box that no workspace row and no unarchived environment names, settled and at least 7 days old.
  • Each run logs and returns what it retired, queued, skipped and deferred. It also writes a retention.run Sentry log line and checks in to the reap-retention cron monitor, so a reaper that stopped is noticed.

How it works

reap-retention every 6 hours (QStash '0 */6 * * *', Sentry.withMonitor)
  cutoff = now - 30d
  rows = asleep vercel rows, not archived, idleSince(cutoff), oldest first, limit 200
    idleSince: created_at < cutoff, sandbox_expires_at null or < cutoff,
               and no boot that is open or started/stopped after cutoff
  for row:
    box running now (provider read)            -> skippedRunning
    retireAndTearDown(row, status = asleep and still idleSince(cutoff), not archived):
      UPDATE ... SET provider_sandbox_id = nextSandboxNameFor(id)
        WHERE id and provider_sandbox_id = old name and condition
      no row matched                           -> skippedChanged
      else deleteSandboxAndDriveLater(old name) -> queue teardown job
    failCloudWorkspace from asleep, still idleSince(cutoff):
      status failed, reason snapshot_expired, sandbox_url null
    pause 250 ms
  if VERCEL_ENV == production:
    boxes = one Vercel listing of every box
    archived backstop: rows archived with deleted_at (or, with none, updated_at) < now - 8d
      found = count; lingering = those whose box is in the listing; alreadyDeleted = found - lingering
      first 20 lingering: reapArchivedCloudWorkspace(delete, archivedAt) or reapUndatedArchivedCloudWorkspace
    orphans: boxes tagged kind=workspace or kind=environment
    keep: status stopped/failed/aborted, created and status-updated > 7 days ago,
          not retired this run, not named by any workspace row (any status)
          or unarchived environment source_ref
    deleteSandboxAndDriveLater for the first 200, 250 ms apart; the rest are counted as deferred
    (before orphans) stranded, first 50:
      provisioning, created_at and updated_at < now - 1d, no boot since   (SQL only; leaves by status)
      failed, coalesce(failed_at, updated_at) < now - 30d, box in listing (a retired name is in no listing)
      retireAndTearDown(row, same predicate); provisioning -> failCloudWorkspace(create_failed); pause 250 ms
  after 200 s a run starts no new work; what is left is counted as deferred

teardown(box) job, retried up to 20 times, 30 s apart
  a row names it -> kept
  stop; a row names it -> kept
  delete (with its snapshots)

The run pauses 250 ms after each teardown it queues: at most 4 a second, far below Vercel's 20 deletions a second per team. The seam takes no delay, so the pacing is in the loop.

Sizing: 4 runs of 200 is 800 a day. Production made a median 156 workspaces a day and at most 725 in the last 30 days, so a normal day clears about 5 times over and even the peak day clears within one day. A backlog takes several runs.

Safety

  • Boot rows are written before Vercel resumes a box, so a wake in flight keeps its box. The reaper uses the latest of the boot rows, sandbox_expires_at and created_at, so when they disagree it deletes later, never earlier.
  • Archive and its 7-day grace period are unchanged. Archived rows are never selected, and the teardown never deletes a box that any row, in any status, still names.
  • Orphan reaping runs only when VERCEL_ENV is production. Every deployment shares one Vercel sandbox project, so a preview or dev database would see production's boxes as orphans.

Drives: a seam for #9847

Main has no workspace drives, and no row names one. Until #9847's drive columns exist, any drive would look orphaned, so this PR reaps boxes only. teardownSandbox uses #9847's route (/api/cloud-workspaces/teardown), payload shape and name, with driveName typed null. #9847's drive-aware version replaces it there.

Existing boxes

@vercel/sandbox 4.0.0 has sandbox.update({ snapshotExpiration }), and the call wakes nothing. It changes the default for snapshots taken from then on; the SDK has no call that changes an existing snapshot's expiry. bun run backfill-snapshot-expiration (packages/sandbox) is a dry run by default. With --apply it updates 2 boxes a second. Not run. Until a box's next stop, the 410 path stays its fallback.

State machine

The model is not edited. The reaper's status write maps to Archive: a Live row (here only asleep) moves to a status that is not Live, and the box is unchanged. Its guard is the boot row, which plays the role of "wake" \notin inflight. The teardown that follows is like Reap, except that it deletes the box. One gap: with billing off there is no boot row, so only the provider read guards a wake in flight. A wake that loses that race finds the row failed, the same as a 410 today.

Production numbers (read-only, 2026-11-18)

  • Asleep vercel workspaces: 857. Asleep and unused for more than 30 days: 0. Within 30 days: 857. The reaper's own SQL agrees: 0 at 30 days, 0 at 7, 518 at 2.
  • By boot rows alone, 188 are unused for more than 7 days, and the oldest last boot is 2026-10-22.
  • A one-off write on 2026-11-16 moved sandbox_expires_at past the last boot on 412 asleep rows. The reaper reaches those no earlier than 2026-12-16; until then Vercel's own expiry still applies to them.

Archived rows (read-only, 2026-11-18)

  • 6,203 archived; 3,553 past grace + 1 day (71 of them with no deleted_at). A random sample of 20 archived rows idle for more than 30 days: all 20 boxes are 404, and the one name checked has no snapshots left. A control box returned 200.

Needs before merge: rebase onto #9847 (whichever lands second)

  • deleteSandboxAndDriveLater: #9847 takes "always queue" (decided).
  • Idle stage: select drive_name and pass row.driveName to retireAndTearDown, and add drive_name = <read> to the condition. With null, #9847's version would clear drive_name and never delete the drive.
  • Stranded stage, drive half: select rows that name a drive as well as a box: failed rows whose box is listed or whose drive_name is not null; stranded provisioning rows as now. Pass row.driveName and add drive_name = <read> to the condition. A failed row whose box is 404 is then still retired (teardown treats a gone box as gone), and leaves the selection because drive_name becomes null. Asleep idle rows already cover case (1). Cases (2) to (4) of #9914's failing property need this.
  • Teardown route payload: take #9847's (driveName nullable, keepDrive).
  • teardown-guard.ts: keep namedSandboxes, take #9847's liveReferences.
  • reap.ts: take #9847's delete stage, and make reapUndatedArchivedCloudWorkspace call retireAndTearDown(row, status archived AND deleted_at null AND updated_at < cutoff).
  • nextSandboxNameFor: import from ./sandbox-name (#9847) instead of ./provision.

Before it runs

  • Create the QStash schedule 0 */6 * * * for /api/cloud-workspaces/jobs/reap-retention in Upstash. It can also be fired once from the "Publish Cloud Workspace Job" workflow.
  • Run the backfill dry, then decide whether to run --apply.

Checks

  • bun test in packages/trpc: 1114 pass. In packages/sandbox: 85 pass.
  • bun run lint; turbo typecheck for trpc, api and sandbox.
  • New tests:
    • retention: "retires each idle workspace, a pause apart", "a box that runs a session now is skipped, not retired", "a row woken or changed after it was read keeps its box", "queues only old, settled boxes no row names, and not one it just retired", "outside production it never looks for orphans", "queues at most a batch of orphans a run and counts the rest", "stops starting work when its time is up and leaves the rest for the next run", "retires the box name while the row is asleep and idle, then fails the row as an expired snapshot", "deletes an archived box still there past its grace through the archive's own delete, by the guard that fits the row", "an archived row unarchived or changed before its delete is counted as skipped"; "retires failed and stranded rows that still name a box, a pause apart, and does not queue their boxes again as orphans", "a failed or stranded row that changed or could not be retired is counted, and the next run tries again", "a stranded provisioning row gives up its box and fails; a failed row only gives up its box", "looks for failed rows only among listed boxes, so a retired name leaves the selection"; the production-only test now also covers archived and stranded rows.
    • reap: "deletes the box of an archived row with no archive date only while it is still archived and unchanged".
    • teardown: "stops, then deletes a box no row names", "keeps a box a row names by the time it would delete it", "a failure requeues the next attempt, and the last attempt gives up", "gives the row a new box name and queues the old box's teardown only when the update matched", "a box without a drive goes through the retried job too, not an inline delete".
    • vercel: "a workspace box's snapshots never expire on Vercel's clock".
    • backfill: "changes only boxes that still expire, paced, and only with --apply".
  • Not run: the TLA+ model (unchanged), and anything against real Vercel.