feat(sandbox): release agent-tools drives every box can rely on (PR A, draft)

Merged
SA
20h ago
+298-86

Draft. PR A of two. This one changes only the release and the drive name a wake mounts; the box side is #9911 (PR B), stacked on this branch.

What it does

  • The release publishes the bundle's agents drive in all 19 regions, and fails before the image, the golden or any row if a region with an environment has no drive, or a drive without claude, codex or pdev.
  • Pruning is safe for rollback and for other databases: every environment row (archived too) moves to the new bundle, the 3 newest released bundles keep their drives in every region, and only --production prunes.
  • A wake mounts the drive of the box's region (not the environment row's), and the golden, probe and real-sandbox boxes mount the drive and check that the agents run from it.

Pseudocode

release:
  needed = {sfo1} ∪ {row.region for every environment row, archived too} ∩ known regions   # B10
  results = publish drive in SANDBOX_REGION_IDS (6 at a time)
  for each needed region: no drive, or manifest lacks claude|codex|pdev -> FAIL               # B4
  kept-previous -> ::warning:: "<region>: <agent> <version> failed; <drive> keeps <sha12> from <parent>"
  image, golden (agentTools: true), probe (agentTools: true; presets resolve into /opt/agents; wake relinks)
  rows: every environment row -> new bundle (archived too)                                    # B1
  if --production:                                                                            # B3
    keep = {drive(row.region, row.bundleSha) for every row}
         ∪ {drive(r, b) for r in 19 regions, b in [new bundle] + 3 newest sandbox_bundles}
    delete agent-tools-* drives not in keep (attached ones are retried next release)
  else: prune skipped

wake (flagged orgs, unchanged otherwise):
  drive = agent-tools-<sandbox.region>-<identity.SUPERSET_BUNDLE_SHA[:12]>                    # B7
create: drive = agent-tools-<environment.region>-<bundle>   (a fork lands in the golden's region, which the row names)

What a customer sees

Nothing changes on any box: the flag still decides who mounts the drive, and every box still has its disk agents. Two visible effects:

  • Workspaces of an archived environment get the current bundle on their next wake, like every other workspace, instead of staying on the bundle the environment had when it was archived.
  • A release can now fail on agents drives, so a box may keep the previous bundle for one more release cycle.

Evidence

  • packages/sandbox bun test: 67 pass. New tests: agentToolsGaps (needed region with missing drive or missing required agent; optional agent and un-needed region ignored), referencedAgentToolsDrives (row drives + recent bundles × regions), the kept-previous warning text with region and version, and available from a publish.
  • packages/trpc bun test: 986 pass. New test: a wake mounts agent-tools-<sandbox.region>-… even when the claim's environment is elsewhere.
  • bun run lint:fix, bun run typecheck (root): clean.
  • Real Vercel: the coordinator reports the real-sandbox check passed on #9911's earlier head (52cb11cb); not re-run on this head (no credentials here).
  • Not run: bun run release, real-sandbox, agent-tools (no credentials; creating sandboxes is not allowed here). No drive publish time was measured (see decision 4).
  • B1 count, production (Neon project Superset, default branch, read-only SELECT): 3 live workspaces in 2 archived environments, on 1 frozen (region, bundle) drive.
    SELECT count(*) AS workspaces, count(DISTINCT e.id) AS archived_environments,
           count(DISTINCT (e.region, e.bundle_sha)) AS frozen_drives
    FROM cloud_workspaces w JOIN environments e ON e.id = w.environment_id
    WHERE w.deleted_at IS NULL AND w.status <> 'archived' AND e.archived_at IS NOT NULL;
    
  • B3, shared project: deploy-production.yml, deploy-paradev-staging.yml, deploy-preview.yml and sandbox.yml all read secrets.VERCEL_SANDBOX_TEAM_ID / VERCEL_SANDBOX_PROJECT_ID. Staging runs in the production GitHub environment with production's DATABASE_URL, so it shares production's rows. Preview runs in the preview environment on a Neon branch of production. The token here cannot list environment secrets (403), so I could not confirm whether preview overrides the project id. The mismatches doc (~812) says everything lives in one sandboxes project. Conclusion: assume one project for every database. Only sandbox.yml's release --production runs in CI; a release from a dev or preview database would see only its own rows, so this PR makes every non-production release skip the prune.

Risks and rollback

  • A release now fails when a needed region's drive is incomplete. That blocks the release, and the image and rows stay on the previous bundle. This is intended, but it means a flaky upstream (claude, codex or pdev download) can hold releases. The earlier drive's version is kept (kept-previous) when the new one fails its check, so this fails only when a region has no earlier drive either.
  • Drives in 19 regions: more builder sandboxes per release (forks of the region's last drive, so only changed agents are fetched).
  • Preview databases (Neon branches) name production's bundles from their branch time; once those bundles leave the newest 3, the production prune deletes their drives. Preview boxes of the flagged org then boot with their disk agents (this PR), or without agents (after PR B).
  • Rollback, step by step:
    1. Revert this PR on main.
    2. Nothing on boxes changes: the next release returns to "regions with an environment" and the old prune.
    3. The old prune deletes the drives this PR kept (all-region and N-1, N-2) on its first run. Before reverting #9911 (PR B) as well, re-run its rollback with drives present (see #9911).

Needs before merge

  • One real-sandbox dispatch on this head.
  • A release on this head (or a dry run of the publish step) to measure the 19-region publish time, and to confirm every region's manifest has claude, codex and pdev.
  • Owner's answer on decisions 3 and 4.

Decisions for the product owner

  1. No drive → open without agents, or fail the open? Recommend: open without agents. Implemented in #9911 (PR B), not here.
  2. Keep disk agents one more release as a fallback? Recommend: no, once N-1 and N-2 drives are kept (this PR). A rollback then has drives to mount.
  3. Should workspaces of archived environments keep working? Recommend: yes. Implemented here by moving archived rows to the new bundle; 3 such workspaces exist in production today.
  4. Cost: drives in all 19 regions per release, vs only regions with environments? Recommend: all 19, because an environment can be created in any region before the next release. Publish time not measured: I have no Vercel credentials here. The release log prints one agent-tools: line per region with a timestamp; the first release on this head gives both numbers (all regions vs the few with environments).

Sequencing

  • #9911 (PR B) is stacked on this branch and merges only after a release with this PR has succeeded and the drives are verified in every region.
  • The API deploys on every push to main (deploy-production.yml), independently of the sandbox release; this PR is safe in either order because it changes no box.

Summary by cubic

Publishes the agent-tools drive in all 19 regions on every release so a box in any region can mount its bundle's agents, and fails the release before the image, golden, or rows change when a region with an environment has no drive or a drive missing claude, codex, or pdev. Previously drives were published only for regions with environments and an incomplete one only warned.

A wake now mounts the drive named by the box's own region instead of its environment row's, and archived environments' workspaces move to the current bundle on their next wake instead of staying on the bundle from when they were archived.

  • Only a --production release prunes drives; it keeps every environment row's drive plus the 3 newest released bundles in all regions, so rollback and mid-release environments keep their mounts.
  • The golden, probe, and real-sandbox mount the drive, and the probe verifies the cloud agent presets run from /opt/agents and the wake relinks them.
  • A release can now fail on a missing or incomplete drive, so a box may keep the previous bundle for one more release cycle.