docs(sandbox): TLA+ model of the workspace drive lifecycle (draft)
A TLA+ model of the workspace drive lifecycle, re-derived from the code at 5c03bd9c and checked again at bf7543b4 (head of #9847's branch, rebased on a newer main). It now includes snapshot expiry and the drive cleanup that follows it. Only .agents/skills/verify-sandbox-drive/ changes; no product code.
The rules (for the product owner)
Your three requirements: nothing a user does is ever lost; an empty /workspace is never shown as their work; a workspace without a drive behaves exactly as today. The model checks them as these rules:
- Nothing is lost. Once a user has work, it stays on the workspace's drive or its box's disk until their deletion is due: delete, then the 7-day grace; or 30 days unused, when the box's snapshot expires and its drive goes too (Satya, 2026-11-18). Work done after an undo is protected again. Harm if broken: commits, changes or installed state disappear.
- Never empty shown as work. We say "ready" only when the box has this workspace's own drive mounted, set up, and holding its work. Harm if broken: the user opens an empty workspace, thinks their work is gone, and may start over on top of it.
- One writer per drive. Two sessions never use one drive at the same time. Harm if broken: the drive reads as corrupt or empty.
- Cleanup never deletes a live workspace's box or drive (neither the cleanup job nor the orphan sweep). Harm if broken: a workspace in use loses its box or its files.
- New: a drive is never deleted while a workspace can still use it. That covers any workspace that names the drive, failed or in its undo window included (they come back through undo). A template's (golden) drive is never deleted. Harm if broken: a workspace the user can still open loses its drive.
- Only creating a workspace gives it a drive; waking it never does. Harm if broken: an old workspace boots onto a drive it never had and hides its own files.
- No drive, no change. A workspace without a drive behaves as today. Harm if broken: workspaces made before drives stop working as they did.
- Never stuck for ever. A live workspace never reaches a state where every open fails and nothing reports it. Harm if broken: "still starting" for ever.
- A new workspace starts on fresh code. Harm if broken: stale code that looks like their branch.
- A passing glitch is not a failure. Harm if broken: a workspace that would come up on the next try shows as broken.
- No drive is left behind for ever. Every drive belongs to a workspace or a pending cleanup, or the sweep deletes it. Harm if broken: unreachable drives bill storage for ever.
- New: an expired workspace's drive is eventually deleted, with the cleanup retried. Harm if broken: drives of workspaces nobody can resume bill storage for ever.
Results at head bf7543b4
| Rule | Holds? |
|---|---|
| Nothing is lost | Yes |
| Never empty shown as work | Yes |
| One writer per drive | Yes |
| Cleanup never deletes a live workspace | Yes |
| Never deleted while a workspace can use it (new) | Yes |
| Only creating gives a drive | Yes |
| No drive, no change | Yes |
| Never stuck for ever | Yes |
| Fresh code for a new workspace | Yes |
| A passing glitch is not a failure | Yes |
| No drive left behind | Yes, if someone runs the sweep (still manual) |
| Expired workspace's drive eventually deleted (new) | No, until the reaper of #9987 lands |
Why the last rule fails. The code deletes an expired workspace's drive only on the 410 that someone opening the workspace gets (a wake, an agent turn or an automation dispatch). It does this through markSandboxUnavailable (wake.ts:262-287): first → failed, then retireAndTearDown where the row is still failed and names that box and drive, then the retried teardown. The drive stays for ever when:
- nobody opens the workspace again;
- the row was already
failedbefore the box expired (no caller reads a failed row); - the 410 path stops between
→ failedand the retire: the function dies, or the retire write throws and is only logged (wake.ts:283-286). The row is then failed and still names its drive, even if the user opens it again; - the row is stuck in
provisioningbecause every delivery of its provision died after the fork (QStash retries twice).
The planned reaper (#9987) must cover all four: take asleep, failed and stranded provisioning rows that still name a drive, and not rely on one call finishing.
Checked against a29d6f17's daily probe, which 5c03bd9c dropped, so you know what the reaper needs:
- the probe closed cases 1 to 3, and with case 4 excluded the rule held (4,217,074 states);
- it could stall: it took the oldest 25 rows a day and skipped a box that answers 404. A failed row whose box is gone keeps its drive name, so 25 such rows would fill every batch for ever.
How the cleanup is modeled. The teardown gives up after 20 attempts. The model gives up only while the drive is still attached, which does not occur after an expiry. Eventual deletion assumes:
- queued teardown and reap jobs run again until they are done;
- a stopped box's drive detaches;
- time passes.
There is no cron in the code at this head, so no cron is assumed.
Also re-checked: every guard and action against the code at 5c03bd9c. #9847 was rebased onto a main that brought M9 (only a wake or a create resumes a box), the failure reasons and the logging changes. No guard changed; only line numbers moved, and these are updated in SKILL.md.
What the guarantee covers, and what it does not
The rules hold for one workspace on one environment, within the bounds below, and on these assumptions:
- The platform behaves as assumed, some of it not yet measured: a
Sessioncall on an ended session fails rather than resumes; a mount can fail and the session starts with nothing mounted; an abort rolls back the root disk and not the drive; Vercel's answer to a mount that names a deleted drive. The model does not rely on Vercel refusing a second read-write mount. - Timing: a lock held past 10 retries (about 6 minutes) fails a provisioning row; the 7-day grace is longer than any provision or wake.
- The sweep is run by someone: it is still a manually dispatched job.
- Git: "untouched" is
HEAD, a cleangit statusand the branch. A user's gitignored file at a path the fetched base tracks could be overwritten on such a boot. Not modeled. - A user working while the first checkout still runs in the background: on a fresh fork that checkout may reclone over it, as on main. Not modeled.
- Snapshot expiry is one event per run. Nothing used the box for 30 days, so no wake, provision or 410 call is in flight on it, and its drive has detached. That
Sandbox.getanswers 410 for it is taken from the code, not measured. - Stops outside the lock (a usage-limit refusal after a re-resume, the meter's cap stops): a stop mounts nothing, so no drive rule depends on it.
- Not modeled at all: the reaper of #9987 (not in the code yet), the sleep state machine (a separate model), the agent-tools drive, home subtrees, Docker on the drive, regions, more than one workspace or environment, promote of the source workspace itself, the deletion of a golden drive itself, a second boot in one session, root-disk work lost by an aborted session (today's behaviour), and boxes no row names.
- Bounds: a bug that needs perturbations from two scenarios at once (for example a bundle downgrade during a re-archive) is not covered.
Bounds
One workspace, one environment, sandbox ids 1 and 2 plus a retired name, drives drive-1 and drive-2, one reprovision, at most 3 sessions, DRIVE_REFUSAL_LIMIT = 3. Eight scenarios; the pass takes about 5 minutes on 8 cores:
| Config | Scenario | Distinct states |
|---|---|---|
Drive.cfg | lifecycle: provision, 2 wakes, archive/unarchive, the grace, the reap, teardown, the sweep, a transient mount failure, a redelivery | 3,035,176 |
Drive-rearchive.cfg | archive, unarchive, archive again | 798,554 |
Drive-race.cfg | two concurrent wakes | 30,752 |
Drive-boot.cfg | bundle change and downgrade, an aborted session, environment entering or leaving drive mode | 382,878 |
Drive-env.cfg | environment change with archive and unarchive | 381,744 |
Drive-twoprov.cfg | two provision jobs for one row (a queued delivery plus the unarchive's, or a QStash duplicate) | 2,267,702 |
Drive-fresh.cfg | a fresh fork whose first fetch fails | 35,990 |
Drive-expiry.cfg | lifecycle with one snapshot expiry and the 410 path, with cleanup fairness | 3,944,490 |
Deeper bounds, run by hand at 1a96ee6b (before expiry was modeled), no error: twoprov with 2 redeliveries (8,780,864 states), fresh with 2 failed fetches, 3 wakes and 4 sessions (113,896), lifecycle with 4 sessions (4,695,594), race with 3 wakes and 2 redeliveries (121,408).
Where the model and the code differ
- A boot is two steps (first layout; everything else); checkout content is abstract; a failed fetch fails the reclone too.
- The resume lock is a set of keys: a wake that finds it held fails, a provision waits or requeues; the 10-retry limit is not modeled.
madeAwakeis the job'sownfield, set by its own transition and cleared by a redelivery.- The persistent-refusal count lives on the box; the drives flag and the golden drive are one variable;
liveReferences's environment checks are not modeled.
How to run
.agents/skills/verify-sandbox-drive/run.sh # every scenario, every rule it lists: must pass
.agents/skills/verify-sandbox-drive/run.sh head # each rule alone at the code (ExpiredDriveDeleted fails)
.agents/skills/verify-sandbox-drive/run.sh bugs # each newly coded guard off: the bug it closed
.agents/skills/verify-sandbox-drive/run.sh broken # every broken config fails its rule
SKILL.md keeps the rule that a product change must not edit the model in the same PR.