feat(cloud): tell clients when a cloud workspace's sandbox is asleep (SUPER-3018)
Links
- Issue: SUPER-3018
- Model-check skill:
.agents/skills/verify-sandbox-asleep/SKILL.md
Summary
- A cloud workspace row now shows when its box is asleep. Every start, extension and stop records
sandbox_expires_atandsandbox_session_id. A QStash watch runs about 5 s after each expiry. The 30-minute reconcile corrects stops we didn't cause and watches that were lost. The desktop sidebar draws a dashed, dimmed cloud for an asleep row. - One status field:
provisioning | awake | asleep | failed | archived(readymeansawake,deletedmeansarchived). This is shipped as expand/contract, with old clients served the old names (details below). - Every asleep write is guarded by session id, which closes a race a TLA+ model found: a stale stop record wrote
asleepover a box another client had just woken.
Why / Context
Vercel sends no event when an idle box stops, so clients had no fresh signal and the sidebar couldn't show that a box had stopped.
How It Works
Expand/contract (no rename in this PR)
The production deploy runs the database step before the API step. A RENAME VALUE would make the running API fail for those minutes. So:
- Migration 0194 (generated by drizzle-kit):
ADD VALUE 'awake','asleep','archived', plus the nullablesandbox_expires_atandsandbox_session_idcolumns the code needs. Neither column has a default, so adding them only changes table metadata. No backfill. - Writes use only the new values.
- Reads accept both spellings through
AWAKE_CLOUD_WORKSPACE_STATUSES,ARCHIVED_CLOUD_WORKSPACE_STATUSESand theis*CloudWorkspaceStatushelpers inpackages/shared/src/cloud-workspace-status.ts. That covers the transitionfromlists, the sleep helpers, archive and reap, and the agent, automation, credential and box-pressure gates. A row still atreadycounts as awake; the null-expiry semantics cover it until the watch or reconcile reclassifies it. - Follow-up PR (TODO on those shared constants): once this API is live, run
UPDATE … ready -> awake, deleted -> archivedand remove the old-spelling reads. The old enum values stay in the type, which is harmless. The backfill must classifyreadyrows from a fullSandbox.list, not from the reconcile's lookback listing. A legacyreadyrow with a null expiry reads as awake. The reconcile lists only boxes whose state changed in the last hour, so a box that stopped weeks ago is never reclassified by the sweep; it is corrected only when someone opens it. That matches today's behaviour, so it isn't a regression.
Legacy status for released clients
Released desktop 1.31.0–2.0.9, CLI 2.0.9, mobile 1.2.2, SDK 0.0.1-alpha.12, and the plugin skills they run open a box only when its status is ready and hide it when it is deleted. cloudWorkspaceStatusFor (packages/trpc/src/router/cloud-workspace/client-status.ts) reads the x-superset-client header. For versions below the floors (desktop/cli 2.0.10, mobile 1.2.3, sdk 0.0.1-alpha.13) it maps awake/asleep to ready and archived to deleted. It applies to every output that carries the status: cloudWorkspace.list, get, create, rename, setTags, and the MCP tools that wrap them. Current clients and web (no header) get the new names, including for rows not yet backfilled.
Removal condition: MINIMUM_DESKTOP_VERSION and MINIMUM_MOBILE_VERSION reach the floors, and api_procedure_called shows no CLI or SDK call below them.
Client checks also accept both spellings, because a mobile OTA update or a desktop dev build can run new code under an old version number. Those builds lose only the asleep glyph.
Session guard
recordSandboxAsleep writes asleep only while the row still records the session that stopped (stillStoppedSession). recordSandboxAwake flips an asleep row only for a new session, because Vercel never resumes a session. Every stop site passes the session it actually stopped:
cloudWorkspace.sleeprestartCloudWorkspace- the spend-cap stops in
meter.ts - the reap's stop stage
- the
queueReapfallback - the watch's confirmed-stopped branch
- the reconcile, against the listing's
currentSessionId
Vercel can return no session id (sessionIdOf returns null when currentSession() throws), and stillStoppedSession(null) turns the guard off. So no caller passes null while the row records a session. Each falls back to the session it read before its provider call: the row's sandboxSessionId (watch, access, sleep, restart, both reap stops) or the boot's vercelSessionId (spend-cap stops). The reconcile skips a stopped listing that has no session id. A test plays the interleaving this closes (watch sees the box stopped, a wake records session 2, the watch writes with no session id) and asserts the row stays awake.
The queueReap fallback (stop at once when the stop stage can't be queued) used to record nothing. That left a future expiry, so an undo inside the grace period showed the stopped box as awake. It now records the stop; on an archived row that writes only the expiry and session id. Sleep writes never throw into the box operation they follow: errors go to Sentry and the regression test is kept.
Also: a CI flake in desktop's terminal-host tests (4c199296a)
Session.dispose() tree-kills the subprocess and PTY pids with SIGKILL, and the terminal-host tests used a fake child with a hardcoded pid 4242 and fake PTY pids. So each test dispose really SIGKILLed whatever process had those pids. On a runner those are the test step's own processes, and turbo test died with exit 137 and no failed test. That happened here (run 43802736018, attempt 1) and twice on shell-style-prod. Session now takes killProcessTree next to spawnProcess, every test that injects a fake child injects a fake kill too, and a test asserts dispose() kills only through the injected function.
What the model verified
.agents/skills/verify-sandbox-asleep/run.sh (TLC, one worker) checks every interleaving of provision, wake, top-up, our stops, the expiry watch, the reconcile, archive and reap.
- Rules:
NeverAsleepWhileRunning,AwakeMeansRunning. - Bounds:
MaxT=6,Timeout=2,MaxGen=3, one workspace, QStash messages late and duplicated. - Result:
Asleep.cfgpasses with 1,624,931 distinct states at depth 32. - Proof it can fail:
Asleep-broken.cfg(CAS = FALSE) failsNeverAsleepWhileRunningin 6 states (ProvisionDone → StopBox → WakeResume → WakeRecord → StopRecord). - Not modelled: closing the boot row (idempotent via
stoppedAt), health, several workspaces, the legacy mapper and old spellings, unarchive.
Testing
bun run lint:fix, thenbiome check .clean.turbo typecheckfor trpc, shared, db, desktop, mobile, cli, sdk, mcp, api: 14/14 tasks.turbo testfor the same packages: trpc 963, shared 1686, desktop 9658, mobile 329, cli 519, api 260 pass, 0 fail.bun run check:i18n: 0 missing in every locale, no catalog changes.sherifclean.bun run check:pluginsclean.- Mutation check: I broke each guard and each stop site in turn (10 mutations), and a test failed every time.
- Migration: applied with
lock_timeout=5s,statement_timeout=60sto a fresh Neon child of production (super-3018-expand-2f986752). The enum becameprovisioning,ready,failed,deleted,awake,asleep,archivedand rows were untouched (ready=741, deleted=5415).cloud_workspacesis 6.7 MB with no job holding it open, so I did not rehearse under load.
Manual QA
- Not verified over CDP. The workspace's dev desktop runs without
RENDERER_REMOTE_DEBUG_PORT, and I did not restart the dev stack in an unattended session. Also, a dev build reports 2.0.9, so it receives legacy values by design. - Preview database (
superset/super-3018-sandbox-asleep-state-2f986752): 0194 applied by the deploy. The enum isprovisioning,ready,failed,deleted,awake,asleep,archived, both columns exist, and rows are untouched (ready=739, deleted=5417). - Not run: these need a signed-in person on the preview API (
api-pr-9800-superset.vercel.app). The agent's sandbox gets 401 there:- (a) our own stop flips a row to
asleepat once, and opening it brings it back toawake; - (b) an idle box flips to
asleepabout 5 s aftersandbox_expires_atvia the watch, the meter settles, and the nudge reaches a client; - (c) a box kept busy across its original expiry stays
awakeand the watch requeues.
- (a) our own stop flips a row to
Known Limitations
packages/trpc/AGENTS.mdrule 10 asks for a schema migration in its own PR. This PR carries 0194 by decision: it only adds values and columns, and the legacy mapper keeps the old output contract for released clients. Reverting the code without the migration is safe.- An installed plugin skill older than this change, used with a CLI at 2.0.10 or later, still polls for
ready. The skill text now accepts both.
Risks / Rollout / Rollback
- Rollout: migration, then API (the normal deploy). The old API keeps working on the expanded enum.
- Rollback: revert the code; the migration stays. Rows the new API wrote as
awake/asleep/archivedread as not ready to the old API until a manualUPDATE … awake/asleep -> ready, archived -> deleted. Run that before (or with) a revert that lands after the new API wrote rows.