feat(sandbox): publish agent-tools drives in parallel, from a fork, per agent

Merged
SA
2d ago
+689-133

What changed

packages/sandbox/src/agent-tools.ts, its test, and the call site in environments/release.ts:

  1. Regions run in parallel. publishAgentToolsDrives runs 6 regions at a time. A region that fails logs the same ::warning::agent-tools: <region> not published… line, and the other regions continue. The bun run agent-tools CLI uses the same path.

  2. A new drive starts as a fork. The release lists agent-tools-<region>-* (sortBy: "name"). It takes the newest drive by createdAt that is not the target and has no sandbox attached, and forks it with drive.fork({ name }). The builder reads that fork's manifest.json from its own read-write mount, so no extra reader sandbox is needed. Then the builder:

    • deletes the old manifest first, so a crash leaves a drive with no manifest, which the next run recreates;
    • reuses any /<name>/<sha>/ that is already there (a directory only exists after an atomic mv from .partial, so if it exists it is complete);
    • fetches only the agents that changed;
    • removes agent directories (and leftover .partial ones) that the new manifest does not name. It skips lost+found;
    • writes the manifest last.

    If there is no earlier drive, or listing or forking fails, it warns, removes any partial fork, and falls back to create-and-write-everything.

  3. Each agent fails alone. A failed download, checksum, unpack or smoke check:

    • keeps the earlier drive's entry for that agent, if the fork still has it and that entry passes the check (kept-previous);
    • otherwise leaves the agent out of the manifest (dropped), and its boxes keep the image's copy.

    An entry is only written to the manifest after its files are in place and its check passed. The publish result lists every agent as written, reused, kept-previous or dropped. The release logs one summary line per region and a ::warning:: for each kept-previous or dropped agent. If no agent is left at all, the builder exits non-zero and that region warns.

  4. Smoke check. Each AGENT_TOOLS row can set check argv (default --version). The builder runs /opt/agents/<name>/<sha>/<bin> <check> from the drive path the box will use. It runs under timeout (30 s, then KILL 5 s later), in an empty temporary HOME and working directory, and with stdin closed.

  5. Manifest shape unchanged: [{ name, sha, bin }], now written compactly. superset-boot reads it with jq. The reader is not changed.

  6. DriveHost gets fork(parent, name). list now returns { name, createdAt, attachedTo }, and prune maps that to names. write now takes (manifest) => script and returns stdout.

No new agent was added. The README row for bun run agent-tools is updated.

Expected saving

  • Before: about 25 s per region, one after another, × 13 regions ≈ 5.5 min per release. Every agent is downloaded and written every time.
  • After (expected, not measured): 13 regions at 6 at a time = 3 waves. A fork takes ~0.5 s (measured, plans/20261116-thin-sandbox-findings.md). With only the changed agents fetched (usually claude, sometimes codex), a wave should be at most today's ~25 s and less when fewer bytes are written, because stop time grows with bytes written. That gives roughly ≤ 75 s, likely 45–60 s. Without a fork (first run, or fork failed), the same 3 waves are still about 75 s.
  • Drives no longer fully rewrite unchanged agents, and one bad agent no longer fails the region's whole drive.

Checks

Ran here:

  • bun test in packages/sandbox: 48 pass, 0 fail (1.8 s). New tests:
    • Fake DriveHost: fork of the newest free same-region drive, with its manifest handed to the builder; fallback with a partial-fork cleanup when fork fails; the four per-agent outcomes and their warnings; 5 regions at concurrency 2 (peak 2), where one failing region stays isolated.
    • Real bash builder script, against a temp dir and a local HTTP server: on a fork, only changed agents are fetched; kept-previous, reused and dropped; removal of unnamed agents, old shas and .partial dirs, with lost+found kept; a custom check argv; a failing check falls back to the earlier version; a hanging check is killed by the timeout.
  • bun run lint:fix: clean.
  • bun run typecheck (whole repo): passes.

Not run:

  • The real Vercel SDK path (vercelDriveHost.fork, list with createdAt, reading the manifest on the builder, running the agents on the builder). This workspace has no real Vercel sandbox credentials. It first runs in the next production release, where any failure only warns and boxes keep the image's agents.
  • The Docker image tests (boot-twice, runner-check, image --local). They were not run because this change does not touch the image, the bundle rootfs or superset-boot.

Risks to watch in the first release

  • The smoke check runs each agent on the builder sandbox (default Vercel image), not the Ubuntu box image. Before this change, agents were never run on the builder. If an agent's --version fails there (a missing library, or a CLI that needs a writable HOME or network for --version), it shows as kept-previous or dropped with a ::warning::. Boxes then use the earlier drive's copy or the image's copy. Nothing breaks, but the warning would mean the check needs a per-agent check override.
  • The previous-drive pick is the newest drive by createdAt that no sandbox has attached. If that drive was left without a manifest by a failed infra run, the fork gives no kept-previous fallbacks. Complete /<name>/<sha>/ directories on it are still reused.

Summary by cubic

Speeds up agent-tools drive publishing and lets each agent fail on its own instead of failing the region's whole drive.

  • Publishes regions six at a time instead of one after another; a failed region logs a ::warning:: and the others continue.
  • Starts each new drive as a fork of the region's newest free earlier drive, so the builder fetches only changed agents, reuses complete existing directories, and writes the manifest last. A failed fork falls back to create-and-write-everything.
  • Each agent keeps the earlier drive's version (kept-previous) or is left out of the manifest (dropped) when its download, checksum, unpack or smoke check fails; if no agent is left, the builder exits non-zero and the region warns.
  • Runs each agent's smoke check (--version by default, 30 s timeout) on the builder before naming it in the manifest.
  • Manifest shape stays [{ name, sha, bin }]; DriveHost gains fork(parent, name), list returns createdAt and attachedTo, and write returns the builder's stdout.

Expected saving: roughly 5.5 min down to ≤75 s per release (not measured).

Things to watch in the first release

  • The smoke check runs on the builder sandbox (default Vercel image), not the box's Ubuntu image; agents never ran on the builder before. A CLI whose --version fails there shows as kept-previous or dropped with a warning, and boxes still use the earlier drive's copy or the image's copy.
  • The real Vercel SDK fork path first runs in the next production release; any failure only warns and boxes keep the image's agents.

Summary by CodeRabbit

  • Improvements
    • Agent-tool releases can reuse the latest earlier drive in each region, updating only tools that have changed.
    • If a tool’s new version fails to download or pass its check, a valid earlier version is retained; otherwise, that tool is omitted from the release.
    • Releases now process multiple regions concurrently and report region-specific results and warnings.