Cloud workspaces v2: the box as a bundle, multi-repository environments, brokered credentials

Merged
SA
2mo ago
+32851-2807

Ten PRs into sandbox-v2, each reviewed by CI and boot tests, merged here as one review point. Records: plans/20260913-sandbox-layout-decisions.md (every decision and who made it), plans/20260913-sandbox-implementation-checklist.md (every item with its evidence), plans/20260914-multi-repo-environments.md.

What changes

  • The box. packages/sandbox: the image, a content-addressed bundle on cdn.superset.sh (runner, steps, assets by sha), superset-boot (root → ubuntu), boot stamps, a start hook the runner sequences and host-service runs. A box updates itself on the next boot when its environment's bundle sha changes; a release publishes, pins, and the box fetches exactly the changed steps.
  • Credentials. Every outbound credential is a firewall header rule the box never sees (Anthropic, OpenAI, GitHub installation token scoped to the workspace's repositories); the environment's variables are pushed live into host-service and inherited by terminals and host-service's own spawns.
  • Environments. Repositories per environment (ordered, the first is what the workspace opens on), config location (.superset/config.json of one of them), organization or personal scope. A lone repository is /workspace; several sit at /workspace/<name>. A promoted environment's repositories are fixed (promote again to change). Desktop dialog with Skip & save / Start agent (onboarding agent), repository and base-branch pills on the new-workspace form; mobile reads the same rows.
  • Gate. Per-port tickets for host-service and the desktop stream (apps/gate was already on main).
  • Schema. One migration, 0115_cloud_environments_repositories, additive with no backfills: the environment scope enum, environment_repositories (environment, repository) and cloud_workspace_repositories (workspace, repository, path) join tables with cascading foreign keys, and bundle_sha, hooks_repository_id, scope, created_by_user_id on environments. Review page: https://app.superset.sh/page/sandbox-v2-migrations-mhf14m

Flagging and blast radius (audited for this PR)

  • Every cloudWorkspace.* and environment.* procedure calls assertCloudAccess, which evaluates the cloud-workspaces PostHog flag server-side. CLI and MCP are unchanged by this branch and go through that gate.
  • Desktop: the sidebar Cloud section, the Cloud host in the device picker and useCloudWorkspaces check the same flag; the environment queries on the new-workspace form fire only once the Cloud host is selected. Mobile's cloud lists check the flag.
  • Code that runs for everyone and what it does without the flag: host-service's terminal env merges an empty managed set; the API samples exactly one Sentry transaction by name (cloud-workspace.provision), everything else stays untraced; the pty socket path only changes when SUPERSET_RUN_DIR is set (sandbox); boot stamps are a timestamp record; the start hook key is parsed and otherwise ignored; the draft store gains a defaulted field.
  • With the flag on but the server refusing (email outside the flag's condition): the sidebar section renders empty, the environments page shows its error state, the new-workspace form toasts "add an environment". No path found that throws into a render.
  • Two pre-existing gaps closed in this branch's last commit: the Settings Environments entry was gated on the v2-app variant rather than the cloud flag, and mobile's environments query was not flag-gated.

Deployment order

Merging deploys the API and runs the migrations. Cloud creates for flagged users then need the new image in the production registry and sandbox:release --production for the internal environment; until then a flagged create fails cleanly (the workspace shows failed). Existing production cloud workspaces from the previous layout are not guaranteed to wake on the new runner; recreate them. Nothing here affects users without the flag.

Owed after deploy

  • Confirm the cloud-workspace.provision transaction arrives in Sentry on the first production create (the API's Sentry is production-only, and #7388 saw no spans from this app).

  • Multi-repository workspaces opening at /workspace with a repository picker in the sidebar: a recorded TODO in plans/20260914-multi-repo-environments.md.

  • A redesign of .superset/config.json for cloud (the flat start and ports keys today): recorded TODO.

Verified (dev, real boxes)

Create, boot, checkout (one and several repositories), private repositories through the firewall with no token on the box, gh inside host-service, stop and wake, update-on-boot with a rollback, the environment dialog and form over CDP, the dev release of the internal golden (twice, on both layouts), the GitHub connect flow in dev. boot-twice and runner-check on every runner change.

https://claude.ai/code/session_01FK6rLzBEUiHHXZWNmpQ2BP

Summary by cubic

Cloud workspaces now boot from a self-updating v2 box and content-addressed bundle instead of the old single-repository sandbox, enabling multi-repository environments, brokered credentials, and personal GitHub access for commits, pushes, and pull requests.

What changed

  • Environments support multiple repositories, organization or personal scope, config-repository start hooks, editing, onboarding agents, and immutable repository sets after promotion.
  • Personal environments are enforced server-side; update calls no longer accept sourceRef or bundleSha.
  • Brokered Anthropic, OpenAI, and GitHub access via firewall rules; environment variables stay in host-service memory and reach terminals and host-service processes.
  • Added GitHub Connections with encrypted token storage, refresh, disconnect, and installation-token fallback when personal OAuth is unavailable; a transient refresh failure no longer removes the connection.
  • The cloud flag now gates the Settings cloud sections, search, and mobile's environments query; environment rows without repositories no longer throw into a render.
  • Added separate gate tickets and diagnostics for host-service and the websockify desktop stream, plus repository and branch selection on desktop and mobile.
  • Sandbox releases run from CI via a production-release dispatch; the image push handles its own registry login.
  • Added migrations 0115 and 0116 for environment repositories and GitHub user connections.

Rollout

  • Deploy the API, run both migrations, publish the production image, and run sandbox:release --production via the workflow's production-release dispatch; flagged creates fail cleanly until the image and bundle are available.
  • Set GH_APP_CLIENT_ID and GH_APP_CLIENT_SECRET to enable personal GitHub connections.
  • The rollout remains behind cloud-workspaces; existing workspaces from the previous layout are not guaranteed to wake and should be recreated.

Summary by CodeRabbit

  • New Features

    • Create and edit environments with repository selection, visibility controls, configuration locations, and optional agent setup.
    • Cloud workspaces support multiple repositories, improved branch selection, and workspace-specific pull requests.
    • Connect personal GitHub accounts for cloud workspace commits, pushes, and pull requests.
    • Sandbox desktops now offer dedicated streaming with boot and readiness diagnostics.
    • Added environment startup commands and refreshed sandbox tooling.

  • Bug Fixes

    • Improved repository handling when environments have no connected repositories.

  • Documentation

    • Expanded sandbox setup, release, troubleshooting, and acceptance documentation.