feat(cloud): tell clients when a cloud workspace's sandbox is asleep (SUPER-3018)

Merged
SA
2d ago
+19546-212

Links

  • Issue: SUPER-3018
  • Model-check skill: .agents/skills/verify-sandbox-asleep/SKILL.md

Summary

  • A cloud workspace row now shows when its box is asleep. Every start, extension and stop records sandbox_expires_at and sandbox_session_id. A QStash watch runs about 5 s after each expiry. The 30-minute reconcile corrects stops we didn't cause and watches that were lost. The desktop sidebar draws a dashed, dimmed cloud for an asleep row.
  • One status field: provisioning | awake | asleep | failed | archived (ready means awake, deleted means archived). This is shipped as expand/contract, with old clients served the old names (details below).
  • Every asleep write is guarded by session id, which closes a race a TLA+ model found: a stale stop record wrote asleep over a box another client had just woken.

Why / Context

Vercel sends no event when an idle box stops, so clients had no fresh signal and the sidebar couldn't show that a box had stopped.

How It Works

Expand/contract (no rename in this PR)

The production deploy runs the database step before the API step. A RENAME VALUE would make the running API fail for those minutes. So:

  • Migration 0194 (generated by drizzle-kit): ADD VALUE 'awake', 'asleep', 'archived', plus the nullable sandbox_expires_at and sandbox_session_id columns the code needs. Neither column has a default, so adding them only changes table metadata. No backfill.
  • Writes use only the new values.
  • Reads accept both spellings through AWAKE_CLOUD_WORKSPACE_STATUSES, ARCHIVED_CLOUD_WORKSPACE_STATUSES and the is*CloudWorkspaceStatus helpers in packages/shared/src/cloud-workspace-status.ts. That covers the transition from lists, the sleep helpers, archive and reap, and the agent, automation, credential and box-pressure gates. A row still at ready counts as awake; the null-expiry semantics cover it until the watch or reconcile reclassifies it.
  • Follow-up PR (TODO on those shared constants): once this API is live, run UPDATE … ready -> awake, deleted -> archived and remove the old-spelling reads. The old enum values stay in the type, which is harmless. The backfill must classify ready rows from a full Sandbox.list, not from the reconcile's lookback listing. A legacy ready row with a null expiry reads as awake. The reconcile lists only boxes whose state changed in the last hour, so a box that stopped weeks ago is never reclassified by the sweep; it is corrected only when someone opens it. That matches today's behaviour, so it isn't a regression.

Legacy status for released clients

Released desktop 1.31.0–2.0.9, CLI 2.0.9, mobile 1.2.2, SDK 0.0.1-alpha.12, and the plugin skills they run open a box only when its status is ready and hide it when it is deleted. cloudWorkspaceStatusFor (packages/trpc/src/router/cloud-workspace/client-status.ts) reads the x-superset-client header. For versions below the floors (desktop/cli 2.0.10, mobile 1.2.3, sdk 0.0.1-alpha.13) it maps awake/asleep to ready and archived to deleted. It applies to every output that carries the status: cloudWorkspace.list, get, create, rename, setTags, and the MCP tools that wrap them. Current clients and web (no header) get the new names, including for rows not yet backfilled.

Removal condition: MINIMUM_DESKTOP_VERSION and MINIMUM_MOBILE_VERSION reach the floors, and api_procedure_called shows no CLI or SDK call below them.

Client checks also accept both spellings, because a mobile OTA update or a desktop dev build can run new code under an old version number. Those builds lose only the asleep glyph.

Session guard

recordSandboxAsleep writes asleep only while the row still records the session that stopped (stillStoppedSession). recordSandboxAwake flips an asleep row only for a new session, because Vercel never resumes a session. Every stop site passes the session it actually stopped:

  • cloudWorkspace.sleep
  • restartCloudWorkspace
  • the spend-cap stops in meter.ts
  • the reap's stop stage
  • the queueReap fallback
  • the watch's confirmed-stopped branch
  • the reconcile, against the listing's currentSessionId

Vercel can return no session id (sessionIdOf returns null when currentSession() throws), and stillStoppedSession(null) turns the guard off. So no caller passes null while the row records a session. Each falls back to the session it read before its provider call: the row's sandboxSessionId (watch, access, sleep, restart, both reap stops) or the boot's vercelSessionId (spend-cap stops). The reconcile skips a stopped listing that has no session id. A test plays the interleaving this closes (watch sees the box stopped, a wake records session 2, the watch writes with no session id) and asserts the row stays awake.

The queueReap fallback (stop at once when the stop stage can't be queued) used to record nothing. That left a future expiry, so an undo inside the grace period showed the stopped box as awake. It now records the stop; on an archived row that writes only the expiry and session id. Sleep writes never throw into the box operation they follow: errors go to Sentry and the regression test is kept.

Also: a CI flake in desktop's terminal-host tests (4c199296a)

Session.dispose() tree-kills the subprocess and PTY pids with SIGKILL, and the terminal-host tests used a fake child with a hardcoded pid 4242 and fake PTY pids. So each test dispose really SIGKILLed whatever process had those pids. On a runner those are the test step's own processes, and turbo test died with exit 137 and no failed test. That happened here (run 43802736018, attempt 1) and twice on shell-style-prod. Session now takes killProcessTree next to spawnProcess, every test that injects a fake child injects a fake kill too, and a test asserts dispose() kills only through the injected function.

What the model verified

.agents/skills/verify-sandbox-asleep/run.sh (TLC, one worker) checks every interleaving of provision, wake, top-up, our stops, the expiry watch, the reconcile, archive and reap.

  • Rules: NeverAsleepWhileRunning, AwakeMeansRunning.
  • Bounds: MaxT=6, Timeout=2, MaxGen=3, one workspace, QStash messages late and duplicated.
  • Result: Asleep.cfg passes with 1,624,931 distinct states at depth 32.
  • Proof it can fail: Asleep-broken.cfg (CAS = FALSE) fails NeverAsleepWhileRunning in 6 states (ProvisionDone → StopBox → WakeResume → WakeRecord → StopRecord).
  • Not modelled: closing the boot row (idempotent via stoppedAt), health, several workspaces, the legacy mapper and old spellings, unarchive.

Testing

  • bun run lint:fix, then biome check . clean.
  • turbo typecheck for trpc, shared, db, desktop, mobile, cli, sdk, mcp, api: 14/14 tasks.
  • turbo test for the same packages: trpc 963, shared 1686, desktop 9658, mobile 329, cli 519, api 260 pass, 0 fail.
  • bun run check:i18n: 0 missing in every locale, no catalog changes. sherif clean. bun run check:plugins clean.
  • Mutation check: I broke each guard and each stop site in turn (10 mutations), and a test failed every time.
  • Migration: applied with lock_timeout=5s, statement_timeout=60s to a fresh Neon child of production (super-3018-expand-2f986752). The enum became provisioning,ready,failed,deleted,awake,asleep,archived and rows were untouched (ready=741, deleted=5415). cloud_workspaces is 6.7 MB with no job holding it open, so I did not rehearse under load.

Manual QA

  • Not verified over CDP. The workspace's dev desktop runs without RENDERER_REMOTE_DEBUG_PORT, and I did not restart the dev stack in an unattended session. Also, a dev build reports 2.0.9, so it receives legacy values by design.
  • Preview database (superset/super-3018-sandbox-asleep-state-2f986752): 0194 applied by the deploy. The enum is provisioning,ready,failed,deleted,awake,asleep,archived, both columns exist, and rows are untouched (ready=739, deleted=5417).
  • Not run: these need a signed-in person on the preview API (api-pr-9800-superset.vercel.app). The agent's sandbox gets 401 there:
    • (a) our own stop flips a row to asleep at once, and opening it brings it back to awake;
    • (b) an idle box flips to asleep about 5 s after sandbox_expires_at via the watch, the meter settles, and the nudge reaches a client;
    • (c) a box kept busy across its original expiry stays awake and the watch requeues.

Known Limitations

  • packages/trpc/AGENTS.md rule 10 asks for a schema migration in its own PR. This PR carries 0194 by decision: it only adds values and columns, and the legacy mapper keeps the old output contract for released clients. Reverting the code without the migration is safe.
  • An installed plugin skill older than this change, used with a CLI at 2.0.10 or later, still polls for ready. The skill text now accepts both.

Risks / Rollout / Rollback

  • Rollout: migration, then API (the normal deploy). The old API keeps working on the expanded enum.
  • Rollback: revert the code; the migration stays. Rows the new API wrote as awake/asleep/archived read as not ready to the old API until a manual UPDATE … awake/asleep -> ready, archived -> deleted. Run that before (or with) a revert that lands after the new API wrote rows.