feat(cloud): add a dry-run-first hygiene script for agent machines

Merged
TE
18d ago
+1082-0

Summary

Persistent agent machines fill their disks: agents build in many worktrees, caches grow, logs pile up. One of our own minis fell to 134 MiB free, and a customer's Cloud machine went unreachable for most of a day with a full disk (#18466). This adds the cleanup engine those machines need, web/services/vms/hygiene/cmux-machine-hygiene. It reports what can be safely reclaimed and, with --apply, reclaims it. Nothing calls it yet. The route, CLI, schedule and UI come in later PRs (plan in manaflow-ai/cmuxterm-hq#2499).

What it reclaims, each item reported with its size:

  • Idle project build output: node_modules, target, .next, .turbo, __pycache__ and similar. Only kondo-style, next to the project's own marker (target needs Cargo.toml), and only when nothing inside changed for 7 days.
  • Package caches (npm, bun, pip, uv, go, yarn, pnpm, cargo tarballs), only when the disk is at least 85% full, since they make the next install warm.
  • Agent logs older than 14 days in ~/.claude/debug, ~/.codex/log and ~/.npm/_logs.
  • Temp entries: the user's own, idle for 7 days.
  • Git worktree records whose directory is already gone (git worktree prune). It never removes a worktree directory.
  • Docker: dangling images, and build cache older than a week. Never volumes.

Dry-run is the default, and --apply needs an explicit --home and --root. Where the script may act at all is an exact allowlist:

  • home is /root, /var/root, /home/NAME or /Users/NAME;
  • every root lies inside home;
  • the temp directory is exactly /tmp, /var/tmp, their /private forms, the user's macOS $TMPDIR (/private/var/folders/X/Y/T, never the rest of that tree), or ~/.cache/tmp, and it never contains a home.

Tests confine the script with CMUX_HYGIENE_SANDBOX: one fresh directory directly inside a temp dir, which home, roots and temp must all sit in. The sandbox must not itself be a mount point, needs a parsable mount list, and --apply under it also needs CMUX_HYGIENE_SANDBOX_APPLY=1. It only narrows what the script may touch.

Every removal goes through one guard, safe_remove, which refuses:

  • symlinks in any path component, re-checked immediately before the rename;
  • mount points, trees holding one, and anything under a mount nested inside home, temp or a root. A same-device bind mount defeats find -xdev and rm's guard, so only the mount list can catch it. The list is read from mount in Linux or macOS format; only \040, \011 and \134 are decoded. If the list is empty, any line fails to parse, or a line has two " on /", nothing is deleted;
  • every deletion when rm can't refuse to cross filesystems (GNU --one-file-system, BSD -x);
  • names with control characters (every scan is NUL-separated);
  • repository roots, and paths holding git-tracked files (literal pathspecs);
  • protected homes: ~/.ssh, ~/.cmux (the daemon's state), ~/.config, credentials, and agent homes and sessions except their log directories;
  • coding-agent scratch directories in temp;
  • anything a process has open, runs from or works in (/proc, or lsof; with neither, nothing is deleted);
  • anything with a socket near its top;
  • anything outside the given roots.

To delete, the script renames the path to .cmux-hygiene-deleting-<name>-<pid> and removes it in the same step, with the guarded rm. That's not a holding area: it only means a run killed midway leaves a clearly named leftover, which the next run finishes.

The rules borrow ideas from Mole (tw93/Mole, GPL-3.0, so no code copied), kondo, cargo-sweep, and glaeda's disk tooling (Apache-2.0).

The script is self-contained bash 3.2. It runs from stdin (bash -s -- ARGS < script), so a later PR can run it through the provider's exec on machines that are already running. Machines never pick up baked files after they're created (docs/cloud-guest-upgrades.md). It's 14 KB, well under the exec limit.

Testing

bun test tests/vm-machine-hygiene.test.ts: 25 pass, 0 fail on b9461cb, under both mawk and gawk 5.2.1 (the CI runners use gawk). The tests were added in four rounds, and each round fails against the code before it:

  • The first 13 were red on eb03ca7 (no script) and green on 401e25f.
  • 5 from the first review (b2730cd) fail against 401e25f:
    • unsafe home, root or temp dir;
    • --apply without --home or --root;
    • a symlinked parent;
    • mount point and contains-mount;
    • newline and glob names.
  • 5 from the second review (c831c74) fail against a42e2d4:
    • exact production allowlists for home and temp;
    • roots outside home, and a sandbox not directly in a temp dir;
    • rm without a mount guard;
    • both mount formats, with odd target names;
    • empty or unparsable mount lists.
  • The third round (55f1495) fails against 1a8ccd4 under gawk. That's the CI shard 3/4 failure: awk -v mangled a backslash path. It also adds:
    • nothing under a nested (bind) mount is removed;
    • a line with two " on /" leaves no mount evidence;
    • the sandbox refuses to start without mount evidence or when it is itself a mount point;
    • --apply under the sandbox needs the test flag.

The macOS per-user .../T acceptance can't be exercised on Linux CI; the refusal side is covered.

Each test builds a throwaway machine under the system temp directory. The helper fails if any reported path lies outside it. docker and mount are stubs on PATH. The tests also cover:

  • dry-run removes nothing;
  • apply keeps source, .git, recent builds, and directories without a marker;
  • a symlinked artifact pointing outside isn't followed;
  • protected homes and sessions are kept;
  • caches are cleared only under pressure;
  • only idle temp entries go;
  • only dead worktree records are pruned;
  • Docker prunes run only with --apply;
  • git-tracked artifacts are kept;
  • a running process inside a project protects it;
  • interrupted leftovers are finished only when their process is gone;
  • the size limit holds.

bun run typecheck: no errors in changed files. The test's in-use process now uses node:child_process, which fixes the Bun.spawn typecheck failure. The only remaining errors are missing workers/presence modules in this sparse checkout. Not run on a real Cloud machine yet; that comes with the route PR.

Changelog

none

Proof

Script and tests only; nothing user-visible until the route and UI land.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added a machine hygiene command that identifies eligible temporary files, old regenerable project artifacts, agent logs, package caches, stale Git worktree records, and dangling Docker images and build cache.
    • The command defaults to dry-run and reports proposed actions, disk usage, and estimated reclaimable space. Applying cleanup requires an explicit home directory and at least one root directory.
    • Category filters let you limit cleanup to selected data types. Package caches are considered only above a configured disk-usage threshold.
  • Safety
    • Cleanup skips paths that are unsafe, protected, in use, tracked by Git, or otherwise ineligible. Failed removals are reported as skipped.