feat(cloud-workspace): workspace drive per environment, flag-gated (draft)
Draft. Stacked on #9915 (the database columns), which lands first. Detail: plans/20261116-workspace-drive.md.
What it does
- A cloud workspace's repository, dependencies and large home folders move to their own Vercel drive. The box's disk keeps everything else, so nothing a person does moves or is lost.
- The box's snapshot then holds only the system: about 2.8 GB instead of about 7.9 GB, so a wake restores less.
- It is off until an organization's
cloud-workspace-drivesflag is on and a release builds that environment's template with a drive. Every other workspace is untouched.
How it works
create (provision)
if the environment has a template drive and the org flag is on:
record drive-<box> on the row; fork the template drive (copy-on-write, ~0.5 s)
under the drive's lock: wait until no box holds the drive, then create the box with it mounted
mark the row ready only if it still names this job's box (else: drop this box, keep the drive)
answer ready only once the boot has laid the drive out; then record "laid out" on the row
boot (on the box)
drive named in sandbox.conf but not mounted, another workspace's, or empty though laid out before
-> refuse; start nothing
lay out: /workspace and the big home folders bind-mounted from the drive;
its state (checkout markers, host.db) kept on the drive too
fresh fork -> fetch the branch as today; a drive with work on it -> never replaced
never re-run with an older boot script that predates the drive
wake
under the drive's lock: a stopped box waits for its drive to be released, remounts, resumes
every other call cannot resume a box; a refusal is retried on the next open,
and only one that cannot heal (3 boots in a row) marks the workspace failed, box and drive kept
stop as today; the drive is released after the box stops
delete + undo
within the 7 days: undo resumes the same box and drive
at the end: the row gets new names in the same write that queues the teardown,
so an undo after it gets a fresh box and drive, never the ones being deleted
teardown: stop, wait for release, delete box, then drive; never what any row still names
What a customer sees
- A faster wake, and their files where they left them. Nothing to learn: the paths are the same.
- If a drive fails to attach, "still starting", then it works on the next open. They never see an empty workspace presented as theirs.
- Nothing they read names a drive. A drive that refuses for good shows as "The workspace did not become ready.", and a drive still flushing at a wake shows as "still starting". A promote of a workspace on a drive that fails, or is refused with the flag off, says "This workspace could not be saved as an environment. Try again later." (translated).
- Create is unchanged when the flag is off. With the flag off in a drive-mode environment, a new workspace clones and installs as it did before drives (slower).
Evidence
Real Vercel, REAL_SANDBOX_DRIVE=1: the same work on a drive box and a plain box (clone, bun install, 4 GB written), five persistence sentinels, an unclean stop, and a workspace created from a drive-mode template.
| Head | Root snapshot, drive vs plain | Wake, drive vs plain | Sentinels kept | Unclean stop / template path |
|---|---|---|---|---|
| e560e42c | 2.58 GB vs 7.86 GB | 5.4 s vs 7.0 s | 5 of 5 | not yet probed |
| e0263ca7 | 2.79 GB vs 7.88 GB | 5 of 5 | ok / ok | |
| 707eabd9 | 2.76 GB vs 7.86 GB | 5 of 5 | ok / ok | |
| 3f7b5ec3 | running |
The wake times are warm: a wake that the snapshot cache does not serve cannot be forced from the check.
The TLA+ model of the drive lifecycle (#9914) checks 10 rules: nothing lost, never empty shown as work, one writer per drive, no teardown of a live workspace, no change without a drive, never stuck, a fresh fork fetches, a glitch is retried, no drive left behind, only create gives a drive. Its author re-derived it from this code at 3f7b5ec3, where 7 held. This head adds the three fixes the model proposed for the other three (FixFailOwnOnly, FixProvLockRetry, FixFetchIfUntouched), each with a unit test or boot-twice case that plays its trace.
- Unmodified
./run.sh(the model's design with those fixes on): every scenario passes. - Unmodified
./run.sh head: still the 3f7b5ec3 code, so those three rules fail until the model is re-derived from this head.
Unit tests replay each trace the model found. boot-twice covers the box side in Docker.
Risks and rollback
- No swap. Cloud boxes have had no swap since #9924, and the drive has none either.
- Kill switch: turn
cloud-workspace-drivesoff. New workspaces then get no drive, promote refuses a drive workspace, and the next release builds the template without one. Workspaces that already have a drive keep it and keep working. The full procedure is in the plan, under "Rollback". - Code revert: workspaces with a drive refuse to boot rather than show an empty
/workspace. Archive them before such a revert. - Costs: drive storage ($0.05/GB-month) plus writes ($0.004/GB). A per-workspace total is not measured yet. A box whose create lost to a delete lives until that delete's 7-day reap.
- Not measured: cold wakes; billing at scale; whether Vercel itself refuses a second mount of an attached drive. The code does not rely on that refusal.
Before internal enable
- #9915 merged and its migrations applied by the normal deploy.
- Real-Vercel check green on the final head (3f7b5ec3 running now).
- The model's author confirms the model edits, and
run.sh headholds on the final head. - One release with the flag on for the internal organization only; watch wake spans and snapshot sizes for a week.
- Run the drive sweep dry in production once.
Before customers
- Schedule the drive sweep and let it delete, after a few clean dry runs.
- Measure drive cost per workspace, and decide how it is billed.
- The items in
docs/cloud-sandbox-considerations.md. - A cold-wake measurement.
- Environments built from
[install](#9282) must setsource_refandgolden_drive_nametogether.
Decisions that are yours
- Flag off in a drive-mode environment: new workspaces get no drive and clone onto the box's disk (slower), rather than being refused. Keep this degrade, or refuse?
- Promote with the flag off: refused for a workspace whose files are on a drive. This reverses the earlier "promote always carries the drive". Confirm.
- Undo after the 7-day grace: a fresh box and drive, because the deletion has completed. Confirm.
- A drive that cannot heal: after 3 boots in a row the workspace shows
failedwith the reason; its box and drive are kept. Keep 3? - Existing workspaces are not moved to drives; only new ones in a drive-mode environment get one. Confirm.
Snapshot expiry deletes the drive
Retention will be ours, in #9987: its reaper, every 6 h, deletes an expired workspace's box, snapshots and drive, instead of relying on Vercel's 30-day snapshotExpiration.
This PR keeps the fallback. When a wake, an agent turn or an automation dispatch gets a 410 for an expired snapshot, markSandboxUnavailable calls retireAndTearDown. That function retires the row's box and drive names, only while the row is failed and still names that box and drive. It then queues the retried teardown, which waits for the drive to detach and never deletes a drive that a row or a golden environment names. The new reaper calls the same function:
// packages/trpc/src/router/cloud-workspace/teardown.ts (export "@superset/trpc/cloud-workspace-teardown")
retireAndTearDown(
row: { id: string; providerSandboxId: string; driveName: string | null },
condition: SQL | undefined, // extra WHERE for the retiring UPDATE, e.g. the status the reaper read
): Promise<boolean> // false: the row no longer matches, or names another box; nothing queued
Checks on this head
This head is #9847 rebased on main (e0f54faf) through #9915 (now 0199), squashed into one commit and resolved by hand. The earlier 33 commits are kept locally as backup/9847-pre-rebase.
Ran:
lint:fix.- Tests: trpc 1160, db 59, api 269, sandbox 89.
- Typecheck: trpc, db, api, sandbox and i18n. The whole-repo run passed 35 of 39 tasks.
check:i18n: the new message is translated in every locale.image --local, and boot-twice withBOOT_TWICE_CA_FILE: all expectations met. This includes "host-service's own env turns the agents' self-updates off, whatever its shell probe finds":DISABLE_AUTOUPDATER,OPENCODE_DISABLE_AUTOUPDATEandPI_SKIP_VERSION_CHECKare now also in the envsuperset-bootstarts host-service with. A chat agent gets that env when host-service's login-shell probe fails.- The model, unmodified at 98667ea2:
./run.sh head, results in the coordinator thread.
Not run:
- Typecheck for desktop, host-service and cli: tsc is killed for running out of memory on this 7 GB box. None of them is changed here.
- The real-Vercel check on this head, and a real 410.