feat(environments): build cloud environments from [install] (SUPER-3017)

Draft
SA
6h ago
+23322-513

Rebased onto main (2026-11-18): the environment page and its Edit and Variables pages moved to /settings/environments/<id> to follow #9707; machine size from #9707 is on the Edit page; the migration is now 0199_environment_builds.

An environment's snapshot (its golden) now follows its base branch. It's rebuilt from [install] on demand, once a day, and when its repositories change. Until now a golden froze on the day it was promoted, so every week new workspaces reinstalled more (Fiveable, thinkfiveable/hours).

Plan: plans/done/20261030-0017-environment-builds.md. Mockup: design/environment-builds/index.html, published at https://app.superset.sh/page/environment-builds-super-3017-mockup-w2u7xq. This replaces #9104, #9108, #9110, #9166, #9167 and #9168.

How a build works

  • environment.rebuild (also the desktop's Trigger new build and superset environments rebuild <env> [--wait]) records a running build and queues a QStash job, /api/environments/build, with maxDuration = 300.
  • The job:
    • forks the golden, or starts from the image when there is none;
    • fetches each repository's base branch and runs the primary repository's [install] as ubuntu, with the environment's variables and SUPERSET_ENV=cloud;
    • strips identity, snapshots, and swaps the golden in a transaction guarded on the old one.
  • Never boots the box, so host-service and [setup] never run in a build. No bundle or image release is needed.
  • Budgets: 45 s for the box and 180 s for [install], inside a 285 s job deadline. A failure keeps the current golden and records the reason (for [install], its last 40 lines).
  • One running build per environment, enforced by a partial unique index (CONFLICT for a second request). A running build older than 10 minutes counts as failed.
  • Refusals: no [install], or a free plan. Builds aren't metered, so the gate is the plan tier.
  • History: the last five goldens are kept. environment.restore makes one active again with its repositories. promote records its golden in the history, and archive deletes every kept golden.
  • Daily: POST /api/environments/jobs/daily-builds builds every eligible environment. It skips free organizations and environments whose last three builds failed.
  • Repository changes: environment.update({ …, startBuild: true }) changes repositories through a build; the environment switches only when the build succeeds. A change that moves a checkout builds from the image. Without startBuild, update refuses as before.
  • Config directory: reads .paradev/config.toml, falling back to .superset/, like the rest of main.

Desktop (to the mockup)

  • Environment page: Overview (repositories, projects, visibility, active build, region), the latest five builds (Daily or Manual, id copyable on hover, the sidebar's grid spinner while running) with Rebuild environment, and the five most recently changed environment variables (All, Add variable).
  • Edit page: name and visibility (settings rows), repositories through the New environment picker in an edit mode, environment variables (the same rows as the full list), and Environment history with Restore. It doesn't show install or setup scripts: they are baked into the box, so the repository's current file isn't what a workspace runs.
  • Settings list: each row has a ⋯ with Edit, Rebuild environment and Archive; build status lives on the environment page only.
  • Left out on purpose: the mockup's "New workspace ↗"; the header has Edit and ⋯ only.

Other

  • environment_builds migration (0186). Only a new table plus one FK to environments, which is small, and no FK to auth tables. It applied in 1.6 s on a Neon branch of production, never a shared or production database.
  • plugins/superset/skills/setup/SKILL.md says when cloud [install] runs and what belongs in [install.cloud] versus [setup.cloud].
  • New strings translated in all 17 locales.
  • Every tRPC change is additive. Desktop and CLI call the new procedures, so the API must deploy before a desktop or CLI release that includes this. Merging this PR does that in order.

Needs doing after merge: create the QStash schedule in Upstash, daily at 09:00 UTC (0 9 * * *), POSTing to /api/environments/jobs/daily-builds.

Verified

  • Unit tests:
    • the build with a fake box and store: success, [install] failing and timing out, box not ready, running out of time before the snapshot, guard lost to a concurrent change, redelivered job, retention;
    • the rules: stale builds, the daily pause, checkout layout moves, removed checkouts, script quoting;
    • the daily loop, the CLI wait loop, [install] parsing.
  • The one-running-build lock, on a Neon branch.
  • Typecheck in db, trpc, cli, api and desktop. Tests: trpc 871, CLI 484, i18n 169, shared 1,539, desktop 9,070. Biome, check:i18n and check:plugins pass.

Not verified

  • A real build on a real box. This cloud workspace has placeholder sandbox credentials, and the PR preview needs a browser login. The test repository's [install] fixture is ParadevAI/acme-demo#8, which needs merging. Steps, against this PR's preview:
    cd packages/cli && export SUPERSET_API_URL=https://api-pr-<this PR>-superset.vercel.app
    bunx cli-framework dev auth login
    bunx cli-framework dev environments create --name "builds e2e" --repo ParadevAI/acme-demo
    bunx cli-framework dev environments rebuild <env> --wait          # succeeded; a workspace on it has ~/.install-marker
    bunx cli-framework dev environments secrets set FAIL_INSTALL --body 1 --environment <env>
    bunx cli-framework dev environments rebuild <env> --wait          # fails with "[install] exited with 3"; golden unchanged
    bunx cli-framework dev environments secrets delete FAIL_INSTALL --environment <env>
    bunx cli-framework dev environments rebuild <env> --wait          # succeeded
    bunx cli-framework dev environments restore <env> <first build id>
    bunx cli-framework dev environments archive <env>
    
    Also time the stop-and-snapshot step against the 285 s deadline.
  • The desktop screens in the running app. There's no desktop to drive over CDP here.

Linear: SUPER-3017


Summary by cubic

An environment's golden is now rebuilt from [install] on its base branch—on demand, once a day, and when its repositories change—instead of freezing when promoted, so new workspaces no longer accumulate install drift (e.g. Fiveable). Replaces #9104, #9108, #9110, #9166, #9167, and #9168.

A build forks the current golden (or starts from the image when there is none), fetches the primary repository's base branch, runs [install], snapshots, and swaps the golden in a transaction guarded on the old one; a failure keeps the current golden and records the last 40 [install] lines. The box never boots, so [setup] and host services never run. One running build per environment, with stale ones counted as failed; the last five goldens are kept, and environment.restore can reactivate one. Free plans and environments without [install] are refused. Daily builds queue for paid organizations and pause after three consecutive failures; repository edits go through a build, switching only when it succeeds.

Desktop gains environment overview, builds, variables, and edit pages under /settings/environments/<id> (the old /environments/<id> redirects). A build ID cell is the copy button (tooltip on hover, "Copied!" after a click), running builds use the sidebar's spinner, and each environment row's ⋯ menu offers Edit, Rebuild environment, and Archive (destructive, after a separator). Builds lists show the latest five; only the daily job reads as Daily, everything else as Manual. The CLI gains environments rebuild [--wait], builds, and restore.

Migration & rollout

  • Adds the environment_builds migration (now 0199). tRPC changes are additive, but the API must deploy before a desktop or CLI release.
  • After merge, create the daily QStash schedule posting to /api/environments/jobs/daily-builds at 09:00 UTC.
  • Not yet verified: a build against a real box and the finished desktop screens in a running app.

Screenshots of the real components: https://app.superset.sh/page/environment-builds-the-ui-as-built-super-3017-db0dqq