#349 Keep automations starting when Daytona's limits are reached

Why

On 2026-10-16 from 12:00 UTC, automation runs in the EU cell failed with "Unable to create or clean up Daytona sandbox …". The real error was hidden:

DaytonaValidationError (400): Total disk limit exceeded. Maximum allowed: 2000GiB.

Stopped sandboxes count toward Daytona's disk limit. Every automation run without a native subscription keeps its sandbox stopped for 24 h. With the 10 GiB snapshot, about 200 runs a day filled the limit (196 stopped sandboxes, 1960 GiB). Archiving them by hand fixed it for now.

What changes

• Paused sandboxes are archived after 30 minutes (setAutoArchiveInterval). Archived sandboxes do not count toward the disk limit and still resume, more slowly.
• Quota rejections are retried, not failed. When Daytona rejects a creation with "… limit exceeded", nothing was created, so the worker:
  • for disk: archives the 20 longest-stopped responder-* sandboxes (stopped > 5 min);
  • waits 5 s, 15 s, 30 s, 60 s, 120 s (about 4 min) and retries the same name;
  • after that fails with DaytonaCapacityError, "Responder's sandbox capacity is temporarily full. Try again in a few minutes." This replaces Daytona's "upgrade your Tier" text.
• Sweep every 10 minutes in the worker: archives sandboxes paused > 30 min (including ones set up before this change), deletes archived and errored responder-* sandboxes older than 24 h, and logs daytona_sandbox_sweep with disk, CPU and memory in use. Deleting does not rely on Daytona's auto-delete covering archived sandboxes.
• Cleanup no longer hides the creation error. A sandbox that never appeared was already reported, so it no longer turns the failure into the generic AggregateError. The remaining aggregate message includes the creation error.

Tests

• sandbox.test.ts: disk-full retry archives only long-stopped sandboxes and reuses the name without waiting for the rejected sandbox to appear; CPU-full waits the full budget, then throws DaytonaCapacityError; paused sandboxes get the archive interval and sandboxes deleted on stop do not; the sweep archives, deletes expired, and logs usage.
• pnpm typecheck, eslint on changed files, and vitest run apps/worker (674 passed).

Follow-ups

• CloudWatch alarm on daytona_sandbox_sweep.diskGiB (hosted repo).
• Consider requeueing a run after DaytonaCapacityError instead of failing it.

---

Summary by cubic

Fixes automation runs failing when the organization's Daytona limits are reached, as happened in the EU cell on 2026-10-16 when stopped sandboxes filled the 2000 GiB disk limit.

• Paused sandboxes are now archived after 30 minutes, so they stop counting toward the disk limit while still being resumable.
• Only Daytona's total CPU, memory and disk limit rejections count as full capacity: the worker archives the longest-stopped responder-* sandboxes (disk only) and retries the same name for about 4 minutes before failing with a clear DaytonaCapacityError. Rate limits keep the normal delete-and-retry cleanup.
• A sweep every 10 minutes archives sandboxes paused longer than 30 minutes, deletes archived, errored or build-failed responder-* sandboxes older than 24 hours, and logs daytona_sandbox_sweep with usage.
• Cleanup failures for a sandbox that never appeared no longer replace the real creation error in the aggregate message.