feat(sandbox): agent CLIs only from the agents drive (PR B, draft)
Draft. PR B of two. PR A (#9929) is merged, and nightly releases with it have published drives in all 19 regions; this PR is rebased onto main. Do not merge until the goldens below have a plan, and #9847 is rebased so both mounts go out in one update({ mounts }).
What it does
- claude, codex, pdev, opencode and pi come only from the agents drive. Their asset rows have no
dest,install-agent-clisis deleted, and thesandbox-agent-tools-driveflag and its plumbing are removed, so every create, wake and promote restart mounts the drive of the box's region. - When the drive cannot be mounted, the open fails. Drive.get, the create and the wake's mounts update retry on 429/5xx. After the last try the create or wake fails the way any box that will not start fails, with the app's existing messages. The customer never learns about drives; region, bundle and drive go only to Sentry and the server log. A boot that finds no manifest at
/opt/agentsis refused before host-service starts. - The box never looks at or touches an older golden's disk copies. Each listed agent's link in
/usr/local/binis replaced with the drive's, so a leftover link cannot win.
Pseudocode
control plane
drive = agent-tools-<region>-<bundle[:12]> region: environment's on create, the sandbox's on wake and promote
mounts = Drive.get(drive) retry 429/5xx x3
missing | attached | still failing -> Sentry {region, bundle, drive}; throw AgentToolsUnavailableError
create: retry 429/5xx x3 { getSandbox(name) ?? create|fork({ ..., mounts }) } no create without mounts
wake: stopped -> update({ mounts: others + agents }) retry 429/5xx x3; still failing -> throw (before resume)
promote: (main's design) snapshot -> golden; promote never boots the source. The caller's restart is a wake (wakeAfterStop),
so it remounts; a failed remount fails that wake, which the caller logs to Sentry, and the golden is still recorded
AgentToolsUnavailableError (message: the existing "The cloud workspace could not start")
wake: treated like SandboxNotReadyError -> existing TIMEOUT "Cloud workspace is still starting" (the desktop retries)
create: failCloudWorkspace reason create_failed -> desktop's existing "Couldn't start cloud workspace";
chat moves into it get the existing "The cloud workspace could not start"
automation: classifyDispatchError -> existing cloud_not_ready
superset-boot link_agents (before host-service)
delete links into /opt/agents that point nowhere
no /opt/agents/manifest.json -> stamp agents.unavailable; refuse # boot.refused, exit 1, host-service never starts
for each manifest entry:
binary on the drive -> ln -sfn into /usr/local/bin/<name> # replaces an old golden's link
else -> rm /usr/local/bin/<name>; stamp agents.missing <name> # no fallback; other agents unaffected
disk copies (/opt/codex, /opt/pdev, media/claude-*) are never read or deleted
goldens
SANDBOX_ROOT_DISK_AGENTS = /opt/codex /opt/pdev media/claude-* steps/install-agent-clis.version
release golden checks + probe (release, real-sandbox) + boot-twice: any of them on the disk -> FAIL
Failure path for a box without the manifest: the existing boot-failure path. refuse stamps boot.refused and exits before start_host_service, so host-service never answers. The control plane's waitForHostService then throws SandboxNotReadyError after HOST_READY_TIMEOUT_MS (60 s):
- create: the provision job marks the workspace
failedand keeps the box stopped for diagnosis (keepForDiagnosis). - wake:
TIMEOUT "Cloud workspace is still starting".
The checkout-failure path was not used. It runs after host-service is up, so it would open a box without agents.
What a customer sees
- Normal: nothing changes. Agents run from
/opt/agents. - Smaller boxes: new boxes and new goldens are 0.79 GB smaller. Existing boxes and goldens keep their old disk copies; those files are inert, and deleting them would not shrink a snapshot (block-level copy, no discard).
- Drive unavailable (missing in the region, attached to a writer, or Vercel's drive API failing after 3 tries):
- opening a stopped workspace answers the existing "Cloud workspace is still starting", and the desktop keeps retrying the wake;
- creating one ends in a
failedworkspace (reasoncreate_failed), which the desktop shows with its existing "Couldn't start cloud workspace"; chats moving into it get the existing "The cloud workspace could not start"; - an automation's run records the existing
cloud_not_ready; - no message names the drive, the region or agents; the reason is only in Sentry (
agent-tools drive not mounted: <reason>, tagssandbox.agent_tools_drive,sandbox.region,sandbox.bundle); - nothing resumes, so nothing is billed as a running box.
- Drive mounted but its manifest missing (an unfinished drive; #9929's release refuses those): the open fails after 60 s as a box that never answered.
- Running session whose host-service died: it is booted again without a remount, because a mount cannot be added to a live session. If that session has no drive, its boots are refused until the box is stopped and opened again, for example with restart.
Trade-off, plainly: a Vercel drive API outage, or a missing drive in a region, now blocks every create and wake in that region. On main, the box would open with its disk agents; in the earlier version of this PR, it would open without agents. Running boxes are unaffected.
Goldens that still hold agents
Every production golden today holds the agents on its disk. No drive-only bundle has shipped yet, and every golden descends from an image that installed them. Deleting the files would not shrink a snapshot (no discard), and every fork carries the bytes, so each such golden has to be built fresh from the image once.
Count (Neon project Superset, default branch, read-only SELECT):
| environments | live workspaces | |
|---|---|---|
golden (source_kind = fork), live | 116 | 820 |
| golden, archived | 25 | 3 |
| image environments (no golden; new boxes come from the new image) | 65 live / 58 archived | 88 / 1 |
All rows show bundle 46b0f9720162. The release moves every row to the newest bundle, so a row's bundle_sha does not say what its golden was built from, and grouping by bundle gives one group. The test therefore has to be a marker on the golden's disk: any of SANDBOX_ROOT_DISK_AGENTS (/opt/codex, /opt/pdev, media/claude-*, steps/install-agent-clis.version). It also catches a golden promoted later from a workspace that still descends from an old golden.
SELECT e.source_kind, e.archived_at IS NOT NULL AS archived, left(e.bundle_sha, 12) AS bundle,
count(DISTINCT e.id) AS environments,
count(w.id) FILTER (WHERE w.deleted_at IS NULL AND w.status <> 'archived') AS live_workspaces
FROM environments e LEFT JOIN cloud_workspaces w ON w.environment_id = e.id
GROUP BY 1, 2, 3;
Paths that make a golden (main, #9282, #9906):
| path | source | forks an existing golden? |
|---|---|---|
main, release.ts internal golden | Sandbox.create from the image | no, fresh every release |
main, promote (environments replace, promoteSandboxToEnvironment) | snapshot of a workspace box | not a fork, but it inherits the workspace's lineage: a workspace forked from an old golden passes the copies on |
#9282, startEnvironmentBuild → rebuildEnvironment (manual, daily, repositories) | kind = moves || sourceKind === "image" ? "fresh" : "incremental"; incremental = { kind: "fork", golden: environment.sourceRef } | yes: every daily and manual build of a fork environment |
#9906, workflow step createBox (build-steps.ts:97) | the same build.kind === "incremental" && sourceKind === "fork" choice | yes, same rule |
Where the fix belongs: #9282 (and #9906 on top of it), not this PR. Main has no build path that forks a golden. I did not edit those branches. The change they need:
- In
rebuildEnvironment(#9282) andcreateBox(#9906), right after an incremental build box is forked, runROOT_DISK_AGENTS_COMMAND(exported by this PR from@superset/shared/sandbox/contract). - If it prints anything: delete that box, create the build box from the image instead (
{ kind: "image", region }), and record the build askind: "fresh". - The fresh golden has no marker, so every later build forks again. It is self-correcting, needs no new column, and costs one wasted fork boot for each affected golden, once.
- One wrinkle: #9282 refuses a build with no
[install]in.paradev/config.toml. A promote-only environment without[install]cannot be rebuilt fresh, so its golden keeps the copies until the owner saves a checkpoint from a workspace made from the image (decision 3).
What one fresh rebuild costs. The nearest measurement is the release's internal golden, built fresh from the image on 8 vCPUs every night: provision, checkout, setup hook (bun install of this monorepo), checks, strip, stop and snapshot. It took 122–145 s in the last four production releases, about 16–19 vCPU-minutes. A customer's build takes that box time plus their own [install] time, which I have no data for (environment_builds only exists in #9282). Each rebuild also writes one new full golden snapshot; #9282 keeps 5 goldens, so storage grows until older ones are pruned.
Rate. All 116 at once would be about 928 vCPUs concurrently, plus 116 clones and dependency installs at once, against GitHub and package registries. Proposal: let #9282's daily job do it, with at most 10 fresh builds running at once and at most 40 started per hour, environments with the most live workspaces first (iad1 and sfo1 hold 558 of the 820). All 116 then finish in about 3 hours of one daily run. If daily runs should stay light, cap fresh rebuilds at 25 a day, which takes 5 days.
Check that a new golden has none.
release.tsgolden checks:no agent CLI on the golden's root disk.probeBox(release probe andreal-sandbox):no agent CLI on the box's root disk, so none in its golden.- boot-twice:
image: no agent CLI on the root disknow uses the same list, including the marker.
Evidence
- Real Vercel:
real-sandboxon359163d2(run 44409700108): every check passes exceptdesktop stream answers RFB through host-service, which fails on main too. Passes include claude, codex and pdev running from the agents drive,the bundle installs no agent CLI on the root disk, andwake: agents linked from the remounted drive. real-sandbox boots the registry'slatestimage, which a main release built with agents on its disk; that image's leftovers (/opt/codex,/opt/pdev, theinstall-agent-clismarker) are logged asinfo. The strictno agent CLI on the box's root diskcheck runs only when the box's image was built from this checkout (a release without--skip-image). - Image (
bun run image --local): 4.89 GB → 4.10 GB. The bundle layer went from 1.52 GB to 721.7 MB. Re-measured on this head: 4.10 GB. - CI on
ced887cc:Boot twicepass (2m47s, with the real checkout), plusLint,Test,Typecheck,SherifandValepass. bun testonced887cc(one test per behaviour):- trpc, 1092 pass, 0 fail:
- pass
agentToolsMounts > retries a drive lookup that failed with 429 or 5xx - pass
agentToolsMounts > a drive that stays unavailable, is missing or a writer holds fails like any box that will not start; region, bundle and drive go only to Sentry(asserts the message is the existing generic one and matches none of /drive|agent|sfo1/) - pass
classifyDispatchError > a box whose agents drive cannot be mounted is a cloud box that is not ready - pass
wakeSandbox > mounts the drive of the box's own region, not its environment's - pass
wakeSandbox > a drive that cannot be mounted fails the wake before the box resumes - pass
wakeSandbox > a mounts update is retried on 5xx, and one that keeps failing fails the wake - pass
provisionSandbox > a create that failed with 5xx is retried with the drive mounted - pass
provisionSandbox > a drive that cannot be mounted fails the create, and no box is made - promote: main's
promoteSandboxToEnvironment > snapshots a running source through its session and leaves its restart to the caller(pass) covers it, since promote never boots a box and the restart is the wake tested above. My own promote test is removed.
- pass
- sandbox, 84 pass, 0 fail.
- shared, 1734 pass, 0 fail.
- host-service, 2921 pass, 23 fail. This PR does not change host-service (its diff against main is empty); the same 23 tests (
session:naming,health.check … sandbox mode) fail here on main's code, and I did not investigate them.
- trpc, 1092 pass, 0 fail:
src/boot-twice.ts, run locally on this head:- pass
every agent on a login shell's PATH runs from the drive - pass
drive: listed agents linked - pass
drive: boot ignores a leftover disk copy - pass
drive: an agent the drive lacks is missing alone, with no fallback to a disk copy - pass
no manifest: boot refused before host-service starts, so the open fails - pass
image: no agent CLI on the root disk(now including theinstall-agent-clismarker) - 3 checkout expectations fail here only: the inner container cannot verify GitHub's TLS behind this workspace's egress CA. CI's
Boot twicejob is the real check.
- pass
src/runner-check.ts: all expectations met.- Lint, typecheck, i18n:
bun run lint:fixclean; typecheck 42/42 (turbo run typecheck --concurrency=2, afterbun installfor a dependency main added; the default concurrency was killed for memory here);bun run check:i18npasses, and this PR no longer changes any catalog (theagentsUnavailablemessage and its translations are gone). - Not run:
bun run release,real-sandbox,agent-tools(no credentials).
Risks and rollback
- Risks:
- A regional drive or Vercel drive API problem is now a regional outage for opens; watch the
agent-tools drive not mountedSentry issue. - A racing second wake whose mounts update fails after the first resumed the box now gets an error instead of being tolerated.
- A missing manifest is reported as a 60 s "still starting" timeout, not by name.
- A regional drive or Vercel drive API problem is now a regional outage for opens; watch the
- Rollback, step by step:
- Keep
sandbox-agent-tools-driveat 100% until this has soaked, so a rollback of only the API to a pre-merge build mounts the drive for everyone through the flag. - Revert this PR on main; the API deploy reads the flag again.
- Run the Sandbox workflow's
production-release: environments move to a bundle withdestrows andinstall-agent-clis. - On each box's next boot, the step runs where its marker is missing or older, and
sync-assetsfetches claude, codex and pdev back onto the disk. A box from an old golden whose marker still matches already has the copies, untouched by this PR. - #9929 keeps the drives of the 3 newest released bundles in every region, so boxes on the reverted bundle still find a drive while they come back.
- Keep
Needs before merge
- #9929 merged (2026-11-18). Nightly
release --productionruns have published 19 regions since; each release fails if a needed region lacks claude, codex or pdev. -
sandbox-agent-tools-driveat 100% and soaked. This PR deletes the flag. - Goldens: the marker check and fresh rebuild in #9282/#9906, and a decision on the rate and on promote-only environments (decisions 3 and 4).
- CI green on
ced887cc. - one
real-sandboxdispatch on this head. - Rebase with #9847 (workspace drive): the two conflict in six files, and the box's mounts must be one composed
update({ mounts }). Whichever lands second rebases; not done here.
Decisions for the product owner
- Should workspaces of archived environments keep working? Recommend: yes, as #9929 does by moving them to the new bundle (3 such workspaces in production).
- Drives in all 19 regions per release, vs only regions with environments? Already all 19 on main. Measured from production-release logs (bundle published → last drive line): 19 regions 49–52 s (3 runs), 13 regions 27 s (1 run), mostly reusing forks of the earlier drive. Recommend: keep all 19; it adds under half a minute to a release.
- Promote-only environments without
[install]cannot be rebuilt fresh. Should we (a) leave their goldens holding about 0.8 GB of inert agents (the image difference this PR measured) until the owner saves a checkpoint from an image-made workspace, (b) tell those owners, or (c) block their saves until they add[install]? Recommend (a) with (b): it costs storage, not correctness. - Rate for the one-time fresh rebuilds (116 live goldens): recommend at most 10 at once and 40 per hour inside the daily job, environments with the most live workspaces first. Alternative: 25 a day.
(Decided by Satya: agents come only from the drive; an open without the drive fails; goldens must not hold agents.)