feat(environments): run environment builds on Vercel Workflow (SUPER-3017)
Stacked on #9282 (base superset/super-3017-environment-builds-all). Draft; do not merge.
What changed
A build was one QStash-delivered function (maxDuration 300, JOB_MS 285 s). Now it is a Vercel Workflow run.
environmentBuildWorkflow("use workflow") and 11"use step"functions inpackages/trpc/src/lib/sandbox/environment-build/build-workflow.tsandbuild-steps.ts. The step bodies are #9282's own functions:prepareBuild,createBuildBox,checkoutScript,succeedBuild/activateGolden,failBuild,pruneGoldens.- No step stays alive while the box works.
startCommandstarts the checkout or[install]detached on the box and returns.readCommandis a short step that reads the command's exit file and the last 40 lines of its output, and saves those lines to the build row. The workflow sleeps 5 s between reads. Stop and snapshot work the same way:stopBox, then a loop ofsnapshotTakenand sleep. vercel.ts: the waitingrunBuildCommandandsnapshotBuildBoxare replaced bystartBuildCommand,readBuildCommand,buildBoxSnapshotId,stopBuildBoxandbuildBoxSnapshotTaken. The box-side scripts are insandbox/build-command.ts.- One product limit:
BUILD_LIMIT_MS= 10 minutes (Satya's call; the old ceiling was about 5), inlimits.ts. Customer message: "The build ran for more than 10 minutes and was stopped". The workflow enforces it while it waits, andtimeouton the box enforces it on the command.STALE_BUILD_MSgoes from 10 min to limit + 10 min (20 min). Without that change, a long build would be marked stale and its box deleted while it was still running. startEnvironmentBuildcallsstart(environmentBuildWorkflow, [buildId], { attributes: { buildId, environmentId } })in place ofpublishCloudWorkspaceJob.apps/api/src/app/api/environments/build/route.ts(the QStash target) is deleted. #9282 is unmerged, so no QStash deliveries to it are in flight.apps/api/next.config.ts:withSentryConfig(withWorkflow(config), …).turbo.jsonc: build outputs includesrc/app/.well-known/workflow/**. Deps:workflow@5.5.0inapps/apiandpackages/trpc. 5.6.x is newer than the repo's 3-dayminimum-release-age. This adds about 730 lines tobun.lock, becauseworkflowdepends on every framework adapter.- DB:
environment_builds.log text(nullable), migration0200_environment_build_log.sql, generated bydrizzle-kit generateafter #9282's0199_environment_builds. Migration checklist: the table is new in #9282 and not in production (0 rows). Adding a nullable column with no default is a metadata-only change. This PR changes no tRPC procedure, sologis not read anywhere yet. ENVIRONMENT_BUILD_FAKE_SANDBOX=1(trpcenv.ts, optional, empty in.env.local.example) makes the steps usefake-build-box.ts, an in-memory box. It is ignored whenNODE_ENV=production. It is not set in any deployed environment, so no GitHub secret was added.- The plan gets decision D10, which supersedes D3.
Workflow, as pseudocode
environmentBuildWorkflow(buildId): # "use workflow"
deadline = now + 10 min # deterministic clock in the workflow VM
try:
sourceRef = beginBuild(buildId) # null -> "skipped"; no [install] -> FatalError
createBox(buildId, sourceRef) # fork the golden read at start, or the image; found by name on retry
prepareBuildDrive(buildId) # no-op: where "lay out the drive before the clone" goes
for key in [checkout, install]:
startCommand(buildId, key, secondsLeft) # detached; mkdir guard -> runs once per key
until (s = readCommand(buildId, key)).exitCode != null: # saves the tail to .log, records commitSha
if now >= deadline: fail(limit)
sleep(5s)
s.exitCode == 124 -> fail(limit); != 0 -> fail("[install] exited with N:\n<tail>")
before = boxSnapshotId(buildId)
stopBox(buildId) # strip identity + build files, stop; no-op if already stopping
until snapshotTaken(buildId, before): deadline check; sleep(5s) # "wait for drive detached" goes here
activateBuild(buildId, sourceRef) or fail("The environment changed during the build…")
catch e:
failBuildAndCleanUp(buildId, FatalError.is(e) ? e.message : "The build stopped unexpectedly")
return "failed"
pruneOldGoldens(buildId)
return "succeeded"
Retry safety:
createBoxis found by name on retry (#9282'sgetSandboxcheck).- A command starts once per key (
mkdiris atomic on the box). Its exit code is written by rename. activateBuildreturnstruewith no second swap when the build is alreadysucceededwith this golden. Its guard is the golden read when the build started, as in #9282, not the one read at activation.failBuildis guarded onstatus = 'running'.
Expected failures are FatalErrors (BuildFailure extends FatalError), so they are not retried. Anything else is retried by the runtime, 3 retries by default; after that comes the one cleanup step.
Secrets never enter the run's event log. Step inputs and outputs are only build ids, the golden ref, a snapshot id and command tails. Each step that needs the network policy or the environment's variables derives them again with prepareBuild.
Drives (out of scope): prepareBuildDrive is the placeholder step for laying out the drive before the clone. The detach wait belongs in the snapshotTaken loop. activateBuild → activateGolden must move the snapshot ref and the golden drive name in one update.
Cancel/supersede: #9282 has neither. With this PR, setting a running build's row to failed stops its run at the next readCommand or stopBox. That step throws FatalError("The build is no longer running"), and the cleanup step deletes the box. This was verified live (below). A future cancel button can write the row, or call getRun(runId).cancel() after finding the run by its buildId attribute.
The five answers
Sources: the docs bundled in the workflow package (node_modules/workflow/docs, the same text as useworkflow.dev, which now redirects to workflow-sdk.dev), https://vercel.com/docs/workflows, /pricing, /concepts, /retention, and https://vercel.com/docs/functions/limitations. Plus what I ran.
1. Limits
- Steps have no limit of their own. A step "must finish within your Vercel Functions duration limit" (vercel.com/docs/workflows/concepts).
- In v5, steps run inline in the generated flow function
/.well-known/workflow/v1/flow. There is no separate step route any more. The builder writessrc/app/.well-known/workflow/v1/config.jsonwith"maxDuration": "max"; I saw it in my localnext build. On Pro or Enterprise with Fluid compute, "max" is 800 s (1800 s "extended maximum", beta). - So a step's ceiling in apps/api is 800 s. That is the same ceiling our own routes have, and it ignores our routes'
maxDurationexports. Our steps take about 0.1–2 s each (measured below). - Replay and orchestration budget: 240 s per invocation, which does not count time inside steps.
- Run duration: "No limit".
sleep: "No limit". Each sleep hop is ≤ 23 h on Vercel Queues. - Events per run: 25,000. Steps per run: 10,000 (default Node VM; QuickJS is exempt).
- Payload: 50 MB per step or run input/output. 2 GB storage per run.
- Run creation: 1,000/s. Events: 200/s per run.
- Our cost: a 10-minute worst case is about 120 polls × ~5 events, so about 600 events. The real 35 s run below wrote 75 events.
2. Availability and pricing (Pro/Enterprise)
- No beta flag; it works on any plan. Experimental parts we don't use: QuickJS,
experimental_hook,experimental_retention,experimentalRoutePrefix. - Price: $0.02 per 1K events, $0.50/GB written, $0.50/GB-month retained, plus normal function compute and Vercel Queues. Hobby includes 50K events; no included Pro quota is documented.
- At 5 s polls, a 3-minute build is about 200 events (about $0.004), and a 10-minute build about 600 events (about $0.012). Sleeping costs no compute.
- Retention on Pro: payloads 7 days after a run ends, metadata 2 days minimum.
- Required project setting: apps/api's Vercel project must have "Enable access to System Environment Variables" on. Workflow picks the Vercel world by
VERCEL_DEPLOYMENT_ID, and without it "every run fails before its first step" (local world,EROFS). I did not check this setting; see "not run".
3. Fit with apps/api
next.config.tscomposes aswithSentryConfig(withWorkflow(config), …). It runs undernext dev(Turbopack) andnext buildwith the Lingui SWC plugin and React compiler on.- Workflow and step functions can live in
packages/trpc. Discovery follows imports from thestart()call. The dev compile reported98 steps, 1 workflow, andmanifest.jsonlistspackages/trpc/src/lib/sandbox/environment-build/build-workflow.ts#environmentBuildWorkflowand all 11 steps. NotranspilePackagesoroutputFileTracingRootwas needed. - The other 87 "steps" are
@vercel/sandbox's own"use step"methods (Command#wait,FileSystem#…). The SDK ships them; they are registered but unused. - bun:
bun testruns the workflow as plain functions, since the directives do nothing without the compiler. The repo'snextruns under Node as before. - Build output changes:
- New functions
ƒ /.well-known/workflow/v1/flowandƒ /.well-known/workflow/v1/webhook/[token], generated intoapps/api/src/app/.well-known/workflow/v1/(self-gitignored). config.jsonholds the queue trigger{"type":"queue/v2beta","topic":"__wkf_workflow_*"}.ƒ /api/environments/buildis gone.proxy.ts's matcher does not touch/.well-known/workflow, so no change is needed there.withWorkflowadded/.swctoapps/api/.gitignore.
- New functions
- Our deploys run
vercel buildin GitHub Actions and thenvercel deploy --prebuilt. The world is chosen at runtime: the compiled bundle hasWORKFLOW_TARGET_WORLD || (VERCEL_DEPLOYMENT_ID ? "vercel" : "local"), so a CI build without a deployment id should not bake in the local world. Only a preview deployment can confirm this.
4. Starting, observing, deployments
start(environmentBuildWorkflow, [buildId], { attributes })returns{ runId: "wrun_…" }as soon as the run is queued. It replaces the QStash publish.- The daily cron route stays: QStash still triggers
queueDailyBuilds. That function callsstartEnvironmentBuild, which callsstart()for each environment; 1,000 runs/s is far above our count. - Mapping a run to a build row:
attributes: { buildId, environmentId }are plaintext and searchable. The attributes came back on the local run:{'buildId': 'aaf83f51-…', 'environmentId': '0000f00d-…'}. No run-id column was added. - Where to look: Vercel dashboard → Project → Observability → Workflows;
npx workflow inspect runs --backend vercel; locally,npx workflow weborinspect. - In-flight runs stay pinned to the deployment that started them. A new deploy does not move them, and a rollback does not stop them. A run whose deployment is deleted never finishes on its own; our stale check (20 min) fails that build row and deletes its box. Fixing a bad run means cancelling it and starting a new one;
deploymentId: "latest"opts in to the newest code.
5. Local dev and tests
- Under
next dev,withWorkflowselects the local world: SQLite in.next/workflow-data, with queue messages delivered back to the dev server over HTTP. - Trap: delivery uses
PORT, and our root.envsetsPORT=3007(next.configloads.envwithoverride: true). To run apps/api on another port, setWORKFLOW_LOCAL_BASE_URL=http://localhost:<port>. - Tests: the SDK only documents Vitest (
@workflow/vitest, withtestTimeout: 60_000). That fits neither bun nor the repo's no-waiting rule. Sobuild-workflow.test.tscalls the workflow as a plain function:- It stubs
workflow.sleepto advance a fake clock (Date.nowis spied). - It stubs
@vercel/sandbox'sSandbox.get/fork/create, so #9282's realcreateBuildBoxruns, and stubs the other box calls. - It wraps every step in a retry loop that acts like the runtime: retry anything that is not a
FatalError, up to 4 attempts.
- It stubs
Product implications (what a customer sees)
- Builds can now take up to 10 minutes, not about 5. An
[install]that took more than 180 s used to fail; now it succeeds. - The new failure message is "The build ran for more than 10 minutes and was stopped". "The build ran out of time before … (285 seconds)" and "[install] ran for more than 180 seconds" no longer appear.
- Same build ids, statuses (
running/succeeded/failed), triggers, kinds and Builds table columns. The tRPC output is unchanged. - The stored log tail (
environment_builds.log) now updates every ~5 s while a build runs. No UI or tRPC reads it yet; that is a follow-up, kept out of the migration PR on purpose. On failure,errorstill has the reason plus the last 40 lines. - A build whose workflow died now shows "The build stopped unexpectedly" after 20 minutes, not 10.
- Builds cost Workflow events: about $0.004 for a typical build, at most about $0.012 (10 minutes ≈ 600 events).
Run for real, locally
- Setup: apps/api under
next dev, local world,ENVIRONMENT_BUILD_FAKE_SANDBOX=1. - Database: this workspace's own Neon branch
br-flat-river-af4vg7vl. It is a child of production, not production. I applied 0186 and 0187 there by hand, then dropped them afterwards. - Data: one synthetic org, environment and repository row (
ParadevAI/acme-demo, the public fixture branchsuper-3017-install-fixture, which has an[install]). These rows were deleted afterwards. - Trigger: a temporary dev-only route, not committed, called the real
startEnvironmentBuild({ trigger: "manual" }).
Trace of a successful run, from the next dev log. Each POST /flow 200 is one invocation that ended at a sleep and was later resumed by a queue message:
[fake-build-box] create envb-aaf83f51502c4c608ffb1191
[fake-build-box] start envb-aaf83f51502c4c608ffb1191/checkout detached
[fake-build-box] read envb-aaf83f51502c4c608ffb1191/checkout: running
[fake-build-box] read envb-aaf83f51502c4c608ffb1191/checkout: running
POST /.well-known/workflow/v1/flow 200 in 2.1s
[fake-build-box] read envb-aaf83f51502c4c608ffb1191/checkout: exit 0
[fake-build-box] start envb-aaf83f51502c4c608ffb1191/install detached
[fake-build-box] read envb-aaf83f51502c4c608ffb1191/install: running
[fake-build-box] read envb-aaf83f51502c4c608ffb1191/install: running
POST /.well-known/workflow/v1/flow 200 in 1066ms
[fake-build-box] read envb-aaf83f51502c4c608ffb1191/install: running
POST /.well-known/workflow/v1/flow 200 in 125ms
[fake-build-box] read envb-aaf83f51502c4c608ffb1191/install: running
POST /.well-known/workflow/v1/flow 200 in 143ms
[fake-build-box] read envb-aaf83f51502c4c608ffb1191/install: running
POST /.well-known/workflow/v1/flow 200 in 137ms
[fake-build-box] read envb-aaf83f51502c4c608ffb1191/install: exit 0
[fake-build-box] stop envb-aaf83f51502c4c608ffb1191
[fake-build-box] snapshot envb-aaf83f51502c4c608ffb1191: not yet
POST /.well-known/workflow/v1/flow 200 in 206ms
[fake-build-box] snapshot envb-aaf83f51502c4c608ffb1191: taken
POST /.well-known/workflow/v1/flow 200 in 596ms
npx workflow inspect runs: wrun_01M7NQGQ959VEGRS8KW7VBV8W4 environmentBuildWorkflow C(ompleted). That is 75 events: 20 steps created/started/completed, 6 wait_created/wait_completed pairs, and the run's own 3. Event tail:
21:45:26.612 step_completed boxSnapshotId
21:45:26.619 step_started stopBox
21:45:26.651 step_completed stopBox
21:45:26.658 step_started snapshotTaken
21:45:26.671 wait_created
21:45:31.692 wait_completed
21:45:31.727 step_started snapshotTaken
21:45:31.740 step_started activateBuild
21:45:32.030 step_completed activateBuild
21:45:32.037 step_started pruneOldGoldens
21:45:32.146 run_completed
The build row afterwards: status succeeded, commit_sha fff…f (from the fake checkout), golden_ref envb-aaf83f51…, log = the last [install] lines, 35.7 s from start to finish. The environment row moved to source_kind fork, source_ref envb-aaf83f51….
Second run, a mid-build cancel: while [install] was running, I set the row to failed / "Cancelled by test". The log shows [workflow-sdk] Encountered FatalError while executing step, bubbling up to parent workflow — FatalError: The build is no longer running, then [fake-build-box] delete envb-cd134288…. inspect steps -d shows the cleanup step got ["cd134288-…", "The build is no longer running"]. So a step's FatalError message reaches the workflow's catch intact on the real runtime, not only in unit tests.
Checks
Ran, all passing:
bun run lint:fix(no fixes left)bun run typecheck(42/42 tasks)bun testinpackages/trpc(869 pass),apps/api(240 pass),packages/db(23 pass)
New tests (bun, no real waiting; the workflow file runs in 0.5 s):
- happy path
- 8-minute
[install](past the old 285 s, inside the 10-minute limit) succeeds [install]exits 2 → failed with its tail, no stop, box deleted- lost answers from
fork,read×2 andactivate, then retried → one box, one swap - build limit hit → limit message, box deleted
- a step that keeps failing → one cleanup, "stopped unexpectedly"
- row marked failed mid-run → stops, box deleted
- environment moved meanwhile → golden kept
- finished build skipped
build-command.test.tsruns the box-side script in real bash: a second start doesn't run it again, and the exit code and tail are read back
I checked that the double-swap test fails when the idempotent branch in activateBuild is removed.
Local next build of apps/api: compiled successfully, and the route list includes the two workflow functions. The command then exited 1 with Next's own SecurityFatalError ("Your version of Next.js is affected by a known security vulnerability"). That is a gate on next@16.4.1 itself and has nothing to do with this change. Note: apps/api is on 16.4.1, not 16.4.5.
Not run, or not run against real services:
- No real Vercel Sandbox. The workspace has placeholder credentials. Detached
runCommand, the box-sidetimeout/mkdirscript on a real box,stop()and snapshot timing were not exercised against Vercel. They were exercised in bash locally and with the fake box. - Real GitHub: only an unauthenticated read of the public fixture's
.superset/config.toml. Because of the synthetic installation row,installationTokenFormay have made one GitHub App token request for an installation that doesn't exist; it fell back to no token, as public repositories do. - No Vercel world: no run on a deployment, no dashboard view, no in-flight run across a deploy. The answers on those come from the docs.
- The runtime's own step retry was not forced live; it is covered by the unit tests' retry loop.
- The 10-minute limit was not hit live; it is covered by unit tests.
- The "Enable access to System Environment Variables" setting on apps/api's Vercel project was not checked.
check:i18nandcheck:pluginswere not run: no Lingui strings or plugins changed.
Preview deployment
Workflow has no supported option to make the generated maxDuration a number: @workflow/next hard-codes 'max', up to the latest 5.1.2. The lowest Vercel CLI that accepts it is 50.31.0. Its @vercel/build-utils 13.8.0 adds || maxDuration === "max", while 50.29.0 / 13.7.0 rejects it, and its changelog reads "Support maxDuration: 'max'" (vercel/vercel#15217). main now has that bump for all workflows (#9980), so this PR no longer changes CI.
Results on head 088222dc (rebased onto #9282 352fd4a9, which is on main 9f04fe88):
| Job | Result |
|---|---|
| Deploy Database (Neon) | pass: 0199_environment_builds, then 0200_environment_build_log |
| Deploy API | pass |
| Deploy Web | pass |
| Deploy Docs | pass |
| Deploy Marketing | pass |
| Deploy Agents Web | pass |
| Lint, Typecheck, Test, Sherif, Version Sync, Vale | pass |
Workflow routes in the API preview (checked on the previous head, with CLI 50.31.0):
- The build lists
ƒ /.well-known/workflow/v1/flowandƒ /.well-known/workflow/v1/webhook/[token]. - The webhook route is deployed:
x-matched-path: /.well-known/workflow/v1/webhook/[token], and its handler answers 404 for an unknown token. - The flow route is not publicly routable:
x-matched-path: /404. That would fit a queue-triggered consumer, but it is unconfirmed without Vercel team access. - No run was started on a preview.
Goldens with agent CLIs on the root disk
createBox runs the root-disk agent check on a forked box. If the check prints anything, the build deletes the fork and starts from the image, with kind=fresh. There is no new column: the next run checks again, so this fixes itself over time.
- Caps: at most 10 fresh builds at once and 40 started per hour. The count and the update are one decision under a Postgres advisory lock (
claimFreshRebuild). - Daily build at the cap: it waits for a slot for up to one hour. After that it forks as before, and the next daily run checks it again.
- Other triggers, or the hourly cap reached: the build forks as before at once.
- Order: the daily run starts the busiest environments first, by the most cloud workspaces created in the last week. Waiting builds take free slots in the order they poll, not strictly busiest first.
- Inlined command:
ROOT_DISK_AGENTS_COMMANDis inlined invercel.tswith a TODO. When #9911 merges, swap it for the import from@superset/shared/sandbox/contract. - Box name: the fresh box gets its own name,
envb-<id>-fresh, so a delete of the fork that Vercel has not finished can't collide with it. Each step reads the name from the build'skind, so a retry still finds the box. Cleanup and stale expiry delete both names. - Not run against real sandboxes. The fake box always reports a clean disk.
Needs before merge
- Swap the inlined
ROOT_DISK_AGENTS_COMMANDfor the export from #9911 once #9911 merges. - Vercel CLI ≥ 50.31.0 in every deploy workflow: done on
mainin #9980. - #9282 rebased onto
main, its migration regenerated as0199_environment_builds; this PR's as0200_environment_build_log. - #9282's own preview database (
br-hidden-hat-afj4fcvw, copied from production on 2026-11-02) fails on main's0195:unsafe use of new value "archived" of enum type cloud_workspace_status. CI replays 0187–0199 in one transaction, and 0195 adds an enum value and then uses it. It needs a reset from production or a fresh branch. I did not touch it. - Turn on "Enable access to System Environment Variables" in apps/api's Vercel project, or confirm it is on.
- One real build on a preview with a real box.