feat(environments): build an environment's golden from [install] (SUPER-3017)

Draft
SA
16d ago
+2174-13

Milestone 1 of plans/20261030-0017-environment-builds.md (#9104). Stacked on the migration PR #9108; merge that first.

What it does

  • environment.rebuild({ environmentId }) records a running build and returns at once. A QStash job (/api/environments/build, maxDuration = 300) then:
    • forks the current golden, or starts from the image when the environment has none;
    • fetches each repository's base branch and runs the primary repository's resolved [install] as ubuntu, with the environment's variables and SUPERSET_ENV=cloud;
    • strips identity, stops, waits for the snapshot, and swaps source_ref in a transaction guarded on the old golden.
  • Budgets: 45 s for the box and 180 s for [install], inside a 285 s deadline for the whole job that the checkout and snapshot share. Exceeding any of them fails the build with the reason.
  • Failure keeps the golden. On any failure the build is marked failed with a readable reason (for [install], its last 40 lines of output), the box is deleted, and the golden is untouched.
  • The box is never booted, so host-service and [setup] never run in a build. No bundle or image release is needed.
  • One build at a time: the database refuses a second running build (partial unique index from #9108), which surfaces as CONFLICT. A running build older than 10 minutes counts as failed and its box is deleted.
  • Refusals:
    • no [install] on the base branch;
    • a free plan: builds are unmetered, so the gate is the plan tier, not authorizeBoot.
  • environment.builds({ environmentId, cursor?, limit? }): newest first, { builds, nextCursor }.
  • environment.restore({ environmentId, buildId }): makes a kept golden active again, with its repositories, with no build. It shares activateGolden with a successful build.
  • Retention: the last five successful goldens are kept; older ones are deleted with their snapshots (settles SUPER-2461 for built goldens).
    • promote records its golden as a build. The golden it replaces is kept for restore when a build row exists for it; otherwise it is deleted as before.
    • archive deletes every kept golden.
  • environment.get / list gain a latestBuild field.
  • CLI: superset environments rebuild <env> [--wait], superset environments builds <env>, superset environments restore <env> <build>.
  • Setup skill: plugins/superset/skills/setup/SKILL.md now says when cloud [install] runs, and that [install.cloud] is for everything not specific to one workspace while [setup.cloud] is for per-workspace state. This plugin ships inside the desktop and CLI builds, not through the marketplace, so there is no plugins publish step; bun run check:plugins passes.
  • Translations: six new server error messages, translated in all 16 locales; bun run check:i18n passes.

tRPC compatibility (.agents/skills/trpc-compat/SKILL.md): every change is additive. Three new procedures, new optional output fields on get/list, and promote and archive keep their input, output and error codes. The new CLI commands call new procedures, so this API must be live before a CLI release that includes them.

Verified

  • Unit tests (fake box, in-memory store): success; fresh starting from the image in the environment's region; [install] cwd; [install] failing and timing out; box not ready in time; guard lost to a concurrent change; redelivered job; retention keeping five and never the active golden. Running out of the job deadline before the snapshot. Rules: build kind, stale builds, the daily pause, checkout script quoting.
  • [install] and [install.cloud] read from TOML (repo-hooks.test.ts).
  • The one-running-build lock, on a dev Neon branch (see #9108).
  • packages/trpc (839 tests), packages/cli (458), packages/i18n (167) pass; trpc, cli and api typecheck; Biome clean; check:i18n and check:plugins pass.

Not verified yet

  • An end-to-end build against a real sandbox. It needs a test repository with [install] on its default branch (none of ours has one), and a local API pointed at a database branch that has #9108. I'll update this description when it's done. Never against a customer environment or the production database.
  • The stop-and-snapshot time on a real box, which decides whether the 285 s deadline leaves enough for it (plan decision D3).

Linear: SUPER-3017


Summary by cubic

Adds environment builds: an async job re-runs the primary repository's [install] (with the environment's variables and SUPERSET_ENV=cloud) on a fork of the current golden—or of the base image when there is none—and swaps the snapshot in on success. Builds never boot the box, so [setup] and host services never run; failures keep the current golden untouched.

New Features

  • environment.rebuild records a running build and returns immediately; a QStash job on /api/environments/build runs it with time budgets and fails the build with the reason when one is exceeded.
  • environment.builds lists builds newest-first with a cursor, and environment.restore makes an earlier golden active again without a build.
  • environment.get/list gain a latestBuild field.
  • CLI: superset environments rebuild <env> [--wait], superset environments builds <env>, and superset environments restore <env> <build>.
  • Retention keeps the five newest successful goldens; promote records its golden as a build, and archive deletes every kept golden.

Migration

  • Builds require Team/Ultra, and the base branch must have [install] in .superset/config.toml; only one build may run per environment.
  • The new CLI commands call new procedures, so the API must be live before a CLI release that includes them; this also needs the environment_builds schema migration.
  • End-to-end builds against a real sandbox are not verified yet, and the snapshot time on a real box decides whether the job deadline holds.

Since the first push: --fresh and --discard-workspace-setup are removed (not needed yet; a build forks the golden, or starts from the image when there is none). The four time budgets, which added up to more than the job's 300 s, are now two plus a job deadline. A successful build and restore share activateGolden.

End-to-end check (to run once ParadevAI/acme-demo#8 is merged)
Against this PR's preview (api-pr-9110-superset.vercel.app), from a checkout of this branch, with a test environment only:

cd packages/cli
export SUPERSET_API_URL=https://api-pr-9110-superset.vercel.app
bunx cli-framework dev auth login                       # the preview has its own sessions
bunx cli-framework dev environments create --name "builds e2e" --repo ParadevAI/acme-demo
bunx cli-framework dev environments rebuild <env> --wait   # expect: succeeded within ~1–2 min (first build is from the image)
bunx cli-framework dev environments builds <env>          # expect: one succeeded build, (active)
# create a cloud workspace on <env>; in it: cat ~/.install-marker  -> "built <sha> at <time>"
bunx cli-framework dev environments secrets set FAIL_INSTALL --body 1 --environment <env>
bunx cli-framework dev environments rebuild <env> --wait   # expect: exit 1, "[install] exited with 3" and the golden unchanged
bunx cli-framework dev environments secrets delete FAIL_INSTALL --environment <env>
bunx cli-framework dev environments rebuild <env> --wait   # expect: succeeded; the first build is now (restorable)
bunx cli-framework dev environments restore <env> <first build id>   # expect: it is (active) again
bunx cli-framework dev environments archive <env>          # deletes every kept golden

Also worth timing: the build's stop-and-snapshot step, which has to fit in what is left of the 285 s deadline.