feat(sandbox): agent CLIs only from the agents drive (PR B, draft)

Draft
SA
2h ago
+540-299

Draft. PR B of two. PR A (#9929) is merged, and nightly releases with it have published drives in all 19 regions; this PR is rebased onto main. Do not merge until the goldens below have a plan, and #9847 is rebased so both mounts go out in one update({ mounts }).

What it does

  • claude, codex, pdev, opencode and pi come only from the agents drive. Their asset rows have no dest, install-agent-clis is deleted, and the sandbox-agent-tools-drive flag and its plumbing are removed, so every create, wake and promote restart mounts the drive of the box's region.
  • When the drive cannot be mounted, the open fails. Drive.get, the create and the wake's mounts update retry on 429/5xx. After the last try the create or wake fails the way any box that will not start fails, with the app's existing messages. The customer never learns about drives; region, bundle and drive go only to Sentry and the server log. A boot that finds no manifest at /opt/agents is refused before host-service starts.
  • The box never looks at or touches an older golden's disk copies. Each listed agent's link in /usr/local/bin is replaced with the drive's, so a leftover link cannot win.

Pseudocode

control plane
  drive = agent-tools-<region>-<bundle[:12]>        region: environment's on create, the sandbox's on wake and promote
  mounts = Drive.get(drive)  retry 429/5xx x3
           missing | attached | still failing -> Sentry {region, bundle, drive}; throw AgentToolsUnavailableError
  create:  retry 429/5xx x3 { getSandbox(name) ?? create|fork({ ..., mounts }) }        no create without mounts
  wake:    stopped -> update({ mounts: others + agents })  retry 429/5xx x3; still failing -> throw (before resume)
  promote: (main's design) snapshot -> golden; promote never boots the source. The caller's restart is a wake (wakeAfterStop),
           so it remounts; a failed remount fails that wake, which the caller logs to Sentry, and the golden is still recorded
  AgentToolsUnavailableError (message: the existing "The cloud workspace could not start")
    wake:       treated like SandboxNotReadyError -> existing TIMEOUT "Cloud workspace is still starting" (the desktop retries)
    create:     failCloudWorkspace reason create_failed -> desktop's existing "Couldn't start cloud workspace";
                chat moves into it get the existing "The cloud workspace could not start"
    automation: classifyDispatchError -> existing cloud_not_ready

superset-boot link_agents (before host-service)
  delete links into /opt/agents that point nowhere
  no /opt/agents/manifest.json -> stamp agents.unavailable; refuse      # boot.refused, exit 1, host-service never starts
  for each manifest entry:
    binary on the drive -> ln -sfn into /usr/local/bin/<name>          # replaces an old golden's link
    else -> rm /usr/local/bin/<name>; stamp agents.missing <name>       # no fallback; other agents unaffected
  disk copies (/opt/codex, /opt/pdev, media/claude-*) are never read or deleted

goldens
  SANDBOX_ROOT_DISK_AGENTS = /opt/codex /opt/pdev media/claude-* steps/install-agent-clis.version
  release golden checks + probe (release, real-sandbox) + boot-twice: any of them on the disk -> FAIL

Failure path for a box without the manifest: the existing boot-failure path. refuse stamps boot.refused and exits before start_host_service, so host-service never answers. The control plane's waitForHostService then throws SandboxNotReadyError after HOST_READY_TIMEOUT_MS (60 s):

  • create: the provision job marks the workspace failed and keeps the box stopped for diagnosis (keepForDiagnosis).
  • wake: TIMEOUT "Cloud workspace is still starting".

The checkout-failure path was not used. It runs after host-service is up, so it would open a box without agents.

What a customer sees

  • Normal: nothing changes. Agents run from /opt/agents.
  • Smaller boxes: new boxes and new goldens are 0.79 GB smaller. Existing boxes and goldens keep their old disk copies; those files are inert, and deleting them would not shrink a snapshot (block-level copy, no discard).
  • Drive unavailable (missing in the region, attached to a writer, or Vercel's drive API failing after 3 tries):
    • opening a stopped workspace answers the existing "Cloud workspace is still starting", and the desktop keeps retrying the wake;
    • creating one ends in a failed workspace (reason create_failed), which the desktop shows with its existing "Couldn't start cloud workspace"; chats moving into it get the existing "The cloud workspace could not start";
    • an automation's run records the existing cloud_not_ready;
    • no message names the drive, the region or agents; the reason is only in Sentry (agent-tools drive not mounted: <reason>, tags sandbox.agent_tools_drive, sandbox.region, sandbox.bundle);
    • nothing resumes, so nothing is billed as a running box.
  • Drive mounted but its manifest missing (an unfinished drive; #9929's release refuses those): the open fails after 60 s as a box that never answered.
  • Running session whose host-service died: it is booted again without a remount, because a mount cannot be added to a live session. If that session has no drive, its boots are refused until the box is stopped and opened again, for example with restart.

Trade-off, plainly: a Vercel drive API outage, or a missing drive in a region, now blocks every create and wake in that region. On main, the box would open with its disk agents; in the earlier version of this PR, it would open without agents. Running boxes are unaffected.

Goldens that still hold agents

Every production golden today holds the agents on its disk. No drive-only bundle has shipped yet, and every golden descends from an image that installed them. Deleting the files would not shrink a snapshot (no discard), and every fork carries the bytes, so each such golden has to be built fresh from the image once.

Count (Neon project Superset, default branch, read-only SELECT):

environmentslive workspaces
golden (source_kind = fork), live116820
golden, archived253
image environments (no golden; new boxes come from the new image)65 live / 58 archived88 / 1

All rows show bundle 46b0f9720162. The release moves every row to the newest bundle, so a row's bundle_sha does not say what its golden was built from, and grouping by bundle gives one group. The test therefore has to be a marker on the golden's disk: any of SANDBOX_ROOT_DISK_AGENTS (/opt/codex, /opt/pdev, media/claude-*, steps/install-agent-clis.version). It also catches a golden promoted later from a workspace that still descends from an old golden.

SELECT e.source_kind, e.archived_at IS NOT NULL AS archived, left(e.bundle_sha, 12) AS bundle,
       count(DISTINCT e.id) AS environments,
       count(w.id) FILTER (WHERE w.deleted_at IS NULL AND w.status <> 'archived') AS live_workspaces
FROM environments e LEFT JOIN cloud_workspaces w ON w.environment_id = e.id
GROUP BY 1, 2, 3;

Paths that make a golden (main, #9282, #9906):

pathsourceforks an existing golden?
main, release.ts internal goldenSandbox.create from the imageno, fresh every release
main, promote (environments replace, promoteSandboxToEnvironment)snapshot of a workspace boxnot a fork, but it inherits the workspace's lineage: a workspace forked from an old golden passes the copies on
#9282, startEnvironmentBuild → rebuildEnvironment (manual, daily, repositories)kind = moves || sourceKind === "image" ? "fresh" : "incremental"; incremental = { kind: "fork", golden: environment.sourceRef }yes: every daily and manual build of a fork environment
#9906, workflow step createBox (build-steps.ts:97)the same build.kind === "incremental" && sourceKind === "fork" choiceyes, same rule

Where the fix belongs: #9282 (and #9906 on top of it), not this PR. Main has no build path that forks a golden. I did not edit those branches. The change they need:

  • In rebuildEnvironment (#9282) and createBox (#9906), right after an incremental build box is forked, run ROOT_DISK_AGENTS_COMMAND (exported by this PR from @superset/shared/sandbox/contract).
  • If it prints anything: delete that box, create the build box from the image instead ({ kind: "image", region }), and record the build as kind: "fresh".
  • The fresh golden has no marker, so every later build forks again. It is self-correcting, needs no new column, and costs one wasted fork boot for each affected golden, once.
  • One wrinkle: #9282 refuses a build with no [install] in .paradev/config.toml. A promote-only environment without [install] cannot be rebuilt fresh, so its golden keeps the copies until the owner saves a checkpoint from a workspace made from the image (decision 3).

What one fresh rebuild costs. The nearest measurement is the release's internal golden, built fresh from the image on 8 vCPUs every night: provision, checkout, setup hook (bun install of this monorepo), checks, strip, stop and snapshot. It took 122–145 s in the last four production releases, about 16–19 vCPU-minutes. A customer's build takes that box time plus their own [install] time, which I have no data for (environment_builds only exists in #9282). Each rebuild also writes one new full golden snapshot; #9282 keeps 5 goldens, so storage grows until older ones are pruned.

Rate. All 116 at once would be about 928 vCPUs concurrently, plus 116 clones and dependency installs at once, against GitHub and package registries. Proposal: let #9282's daily job do it, with at most 10 fresh builds running at once and at most 40 started per hour, environments with the most live workspaces first (iad1 and sfo1 hold 558 of the 820). All 116 then finish in about 3 hours of one daily run. If daily runs should stay light, cap fresh rebuilds at 25 a day, which takes 5 days.

Check that a new golden has none.

  • release.ts golden checks: no agent CLI on the golden's root disk.
  • probeBox (release probe and real-sandbox): no agent CLI on the box's root disk, so none in its golden.
  • boot-twice: image: no agent CLI on the root disk now uses the same list, including the marker.

Evidence

  • Real Vercel: real-sandbox on 359163d2 (run 44409700108): every check passes except desktop stream answers RFB through host-service, which fails on main too. Passes include claude, codex and pdev running from the agents drive, the bundle installs no agent CLI on the root disk, and wake: agents linked from the remounted drive. real-sandbox boots the registry's latest image, which a main release built with agents on its disk; that image's leftovers (/opt/codex, /opt/pdev, the install-agent-clis marker) are logged as info. The strict no agent CLI on the box's root disk check runs only when the box's image was built from this checkout (a release without --skip-image).
  • Image (bun run image --local): 4.89 GB → 4.10 GB. The bundle layer went from 1.52 GB to 721.7 MB. Re-measured on this head: 4.10 GB.
  • CI on ced887cc: Boot twice pass (2m47s, with the real checkout), plus Lint, Test, Typecheck, Sherif and Vale pass.
  • bun test on ced887cc (one test per behaviour):
    • trpc, 1092 pass, 0 fail:
      • pass agentToolsMounts > retries a drive lookup that failed with 429 or 5xx
      • pass agentToolsMounts > a drive that stays unavailable, is missing or a writer holds fails like any box that will not start; region, bundle and drive go only to Sentry (asserts the message is the existing generic one and matches none of /drive|agent|sfo1/)
      • pass classifyDispatchError > a box whose agents drive cannot be mounted is a cloud box that is not ready
      • pass wakeSandbox > mounts the drive of the box's own region, not its environment's
      • pass wakeSandbox > a drive that cannot be mounted fails the wake before the box resumes
      • pass wakeSandbox > a mounts update is retried on 5xx, and one that keeps failing fails the wake
      • pass provisionSandbox > a create that failed with 5xx is retried with the drive mounted
      • pass provisionSandbox > a drive that cannot be mounted fails the create, and no box is made
      • promote: main's promoteSandboxToEnvironment > snapshots a running source through its session and leaves its restart to the caller (pass) covers it, since promote never boots a box and the restart is the wake tested above. My own promote test is removed.
    • sandbox, 84 pass, 0 fail.
    • shared, 1734 pass, 0 fail.
    • host-service, 2921 pass, 23 fail. This PR does not change host-service (its diff against main is empty); the same 23 tests (session: naming, health.check … sandbox mode) fail here on main's code, and I did not investigate them.
  • src/boot-twice.ts, run locally on this head:
    • pass every agent on a login shell's PATH runs from the drive
    • pass drive: listed agents linked
    • pass drive: boot ignores a leftover disk copy
    • pass drive: an agent the drive lacks is missing alone, with no fallback to a disk copy
    • pass no manifest: boot refused before host-service starts, so the open fails
    • pass image: no agent CLI on the root disk (now including the install-agent-clis marker)
    • 3 checkout expectations fail here only: the inner container cannot verify GitHub's TLS behind this workspace's egress CA. CI's Boot twice job is the real check.
  • src/runner-check.ts: all expectations met.
  • Lint, typecheck, i18n: bun run lint:fix clean; typecheck 42/42 (turbo run typecheck --concurrency=2, after bun install for a dependency main added; the default concurrency was killed for memory here); bun run check:i18n passes, and this PR no longer changes any catalog (the agentsUnavailable message and its translations are gone).
  • Not run: bun run release, real-sandbox, agent-tools (no credentials).

Risks and rollback

  • Risks:
    • A regional drive or Vercel drive API problem is now a regional outage for opens; watch the agent-tools drive not mounted Sentry issue.
    • A racing second wake whose mounts update fails after the first resumed the box now gets an error instead of being tolerated.
    • A missing manifest is reported as a 60 s "still starting" timeout, not by name.
  • Rollback, step by step:
    1. Keep sandbox-agent-tools-drive at 100% until this has soaked, so a rollback of only the API to a pre-merge build mounts the drive for everyone through the flag.
    2. Revert this PR on main; the API deploy reads the flag again.
    3. Run the Sandbox workflow's production-release: environments move to a bundle with dest rows and install-agent-clis.
    4. On each box's next boot, the step runs where its marker is missing or older, and sync-assets fetches claude, codex and pdev back onto the disk. A box from an old golden whose marker still matches already has the copies, untouched by this PR.
    5. #9929 keeps the drives of the 3 newest released bundles in every region, so boxes on the reverted bundle still find a drive while they come back.

Needs before merge

  • #9929 merged (2026-11-18). Nightly release --production runs have published 19 regions since; each release fails if a needed region lacks claude, codex or pdev.
  • sandbox-agent-tools-drive at 100% and soaked. This PR deletes the flag.
  • Goldens: the marker check and fresh rebuild in #9282/#9906, and a decision on the rate and on promote-only environments (decisions 3 and 4).
  • CI green on ced887cc.
  • one real-sandbox dispatch on this head.
  • Rebase with #9847 (workspace drive): the two conflict in six files, and the box's mounts must be one composed update({ mounts }). Whichever lands second rebases; not done here.

Decisions for the product owner

  1. Should workspaces of archived environments keep working? Recommend: yes, as #9929 does by moving them to the new bundle (3 such workspaces in production).
  2. Drives in all 19 regions per release, vs only regions with environments? Already all 19 on main. Measured from production-release logs (bundle published → last drive line): 19 regions 49–52 s (3 runs), 13 regions 27 s (1 run), mostly reusing forks of the earlier drive. Recommend: keep all 19; it adds under half a minute to a release.
  3. Promote-only environments without [install] cannot be rebuilt fresh. Should we (a) leave their goldens holding about 0.8 GB of inert agents (the image difference this PR measured) until the owner saves a checkpoint from an image-made workspace, (b) tell those owners, or (c) block their saves until they add [install]? Recommend (a) with (b): it costs storage, not correctness.
  4. Rate for the one-time fresh rebuilds (116 live goldens): recommend at most 10 at once and 40 per hour inside the daily job, environments with the most live workspaces first. Alternative: 25 a day.

(Decided by Satya: agents come only from the drive; an open without the drive fails; goldens must not hold agents.)