feat(sandbox): publish agent-tools drives in parallel, from a fork, per agent
What changed
packages/sandbox/src/agent-tools.ts, its test, and the call site in environments/release.ts:
-
Regions run in parallel.
publishAgentToolsDrivesruns 6 regions at a time. A region that fails logs the same::warning::agent-tools: <region> not published…line, and the other regions continue. Thebun run agent-toolsCLI uses the same path. -
A new drive starts as a fork. The release lists
agent-tools-<region>-*(sortBy: "name"). It takes the newest drive bycreatedAtthat is not the target and has no sandbox attached, and forks it withdrive.fork({ name }). The builder reads that fork'smanifest.jsonfrom its own read-write mount, so no extra reader sandbox is needed. Then the builder:- deletes the old manifest first, so a crash leaves a drive with no manifest, which the next run recreates;
- reuses any
/<name>/<sha>/that is already there (a directory only exists after an atomicmvfrom.partial, so if it exists it is complete); - fetches only the agents that changed;
- removes agent directories (and leftover
.partialones) that the new manifest does not name. It skipslost+found; - writes the manifest last.
If there is no earlier drive, or listing or forking fails, it warns, removes any partial fork, and falls back to create-and-write-everything.
-
Each agent fails alone. A failed download, checksum, unpack or smoke check:
- keeps the earlier drive's entry for that agent, if the fork still has it and that entry passes the check (
kept-previous); - otherwise leaves the agent out of the manifest (
dropped), and its boxes keep the image's copy.
An entry is only written to the manifest after its files are in place and its check passed. The publish result lists every agent as
written,reused,kept-previousordropped. The release logs one summary line per region and a::warning::for each kept-previous or dropped agent. If no agent is left at all, the builder exits non-zero and that region warns. - keeps the earlier drive's entry for that agent, if the fork still has it and that entry passes the check (
-
Smoke check. Each
AGENT_TOOLSrow can setcheckargv (default--version). The builder runs/opt/agents/<name>/<sha>/<bin> <check>from the drive path the box will use. It runs undertimeout(30 s, then KILL 5 s later), in an empty temporaryHOMEand working directory, and with stdin closed. -
Manifest shape unchanged:
[{ name, sha, bin }], now written compactly.superset-bootreads it withjq. The reader is not changed. -
DriveHostgetsfork(parent, name).listnow returns{ name, createdAt, attachedTo }, and prune maps that to names.writenow takes(manifest) => scriptand returns stdout.
No new agent was added. The README row for bun run agent-tools is updated.
Expected saving
- Before: about 25 s per region, one after another, × 13 regions ≈ 5.5 min per release. Every agent is downloaded and written every time.
- After (expected, not measured): 13 regions at 6 at a time = 3 waves. A fork takes ~0.5 s (measured,
plans/20261116-thin-sandbox-findings.md). With only the changed agents fetched (usually claude, sometimes codex), a wave should be at most today's ~25 s and less when fewer bytes are written, because stop time grows with bytes written. That gives roughly ≤ 75 s, likely 45–60 s. Without a fork (first run, or fork failed), the same 3 waves are still about 75 s. - Drives no longer fully rewrite unchanged agents, and one bad agent no longer fails the region's whole drive.
Checks
Ran here:
bun testinpackages/sandbox: 48 pass, 0 fail (1.8 s). New tests:- Fake
DriveHost: fork of the newest free same-region drive, with its manifest handed to the builder; fallback with a partial-fork cleanup when fork fails; the four per-agent outcomes and their warnings; 5 regions at concurrency 2 (peak 2), where one failing region stays isolated. - Real bash builder script, against a temp dir and a local HTTP server: on a fork, only changed agents are fetched; kept-previous, reused and dropped; removal of unnamed agents, old shas and
.partialdirs, withlost+foundkept; a customcheckargv; a failing check falls back to the earlier version; a hanging check is killed by the timeout.
- Fake
bun run lint:fix: clean.bun run typecheck(whole repo): passes.
Not run:
- The real Vercel SDK path (
vercelDriveHost.fork,listwithcreatedAt, reading the manifest on the builder, running the agents on the builder). This workspace has no real Vercel sandbox credentials. It first runs in the next production release, where any failure only warns and boxes keep the image's agents. - The Docker image tests (
boot-twice,runner-check,image --local). They were not run because this change does not touch the image, the bundle rootfs orsuperset-boot.
Risks to watch in the first release
- The smoke check runs each agent on the builder sandbox (default Vercel image), not the Ubuntu box image. Before this change, agents were never run on the builder. If an agent's
--versionfails there (a missing library, or a CLI that needs a writable HOME or network for--version), it shows askept-previousordroppedwith a::warning::. Boxes then use the earlier drive's copy or the image's copy. Nothing breaks, but the warning would mean the check needs a per-agentcheckoverride. - The previous-drive pick is the newest drive by
createdAtthat no sandbox has attached. If that drive was left without a manifest by a failed infra run, the fork gives no kept-previous fallbacks. Complete/<name>/<sha>/directories on it are still reused.
Summary by cubic
Speeds up agent-tools drive publishing and lets each agent fail on its own instead of failing the region's whole drive.
- Publishes regions six at a time instead of one after another; a failed region logs a
::warning::and the others continue. - Starts each new drive as a fork of the region's newest free earlier drive, so the builder fetches only changed agents, reuses complete existing directories, and writes the manifest last. A failed fork falls back to create-and-write-everything.
- Each agent keeps the earlier drive's version (
kept-previous) or is left out of the manifest (dropped) when its download, checksum, unpack or smoke check fails; if no agent is left, the builder exits non-zero and the region warns. - Runs each agent's smoke check (
--versionby default, 30 s timeout) on the builder before naming it in the manifest. - Manifest shape stays
[{ name, sha, bin }];DriveHostgainsfork(parent, name),listreturnscreatedAtandattachedTo, andwritereturns the builder's stdout.
Expected saving: roughly 5.5 min down to ≤75 s per release (not measured).
Things to watch in the first release
- The smoke check runs on the builder sandbox (default Vercel image), not the box's Ubuntu image; agents never ran on the builder before. A CLI whose
--versionfails there shows askept-previousordroppedwith a warning, and boxes still use the earlier drive's copy or the image's copy. - The real Vercel SDK fork path first runs in the next production release; any failure only warns and boxes keep the image's agents.
Summary by CodeRabbit
- Improvements
- Agent-tool releases can reuse the latest earlier drive in each region, updating only tools that have changed.
- If a tool’s new version fails to download or pass its check, a valid earlier version is retained; otherwise, that tool is omitted from the release.
- Releases now process multiple regions concurrently and report region-specific results and warnings.