feat(worktree-care): machines clean up after their agents (archive, reclaim, restore on reopen)

Merged
TE
16d ago
+4042-17

Machines that clean up after their agents, as a product feature (dogfood first). Today huge-blue sits at 90–94% disk, cosmic-gray holds ~100 cmux worktrees, and every agent leaves node_modules/target/ behind. This PR adds a worktree lifecycle reconciler to acpmux, so every machine running cmux cleans itself up.

What it does

New crate cmux-tui/crates/cmux-worktree-care:

  • inventory: for every linked worktree of the repositories under the roots, it records branch, HEAD, changed and untracked files, unpushed commits, ignored build output vs ignored data, sparse patterns, submodules, and last activity (HEAD's newest reflog entry, not the file mtime that git gc rewrites, plus the index and changed files). Squash-merged work counts as finished: merging HEAD into the base changes nothing. Extra bases: git config --add worktreeCare.base origin/feat-cmux-next.
  • policy: pure verdicts, unit-tested. Order of guarantees:
    1. Never touched: locked worktrees, a rebase/merge/bisect under way, the folder of an open session (from every acpmux state folder on the machine: ~/.acpmux*, every ~/.cmux/chief/*/acpmux), the cwd of any process.
    2. Never lost: a worktree no archive can keep whole (a nested repo, a changed submodule, over 512 MiB of data) is only trimmed.
    3. Recent work is left alone (6 h / 3 h / 1 h by pressure).
    4. Then reclaim: finished work right away; pushed and clean after 72 h / 24 h / 6 h; work that exists only here goes into an archive after 7 d / 72 h / 24 h. Waits shrink at 80% (high) and 90% (critical) disk used.
  • archive: the worktree's index and files become commits shaped like git stash, under refs/cmux/archive/<id>, plus a bundle of whatever the remotes lack and a manifest in ~/.cmux/worktree-archive. Then git worktree remove. Status and HEAD are read again right before removal, and any change stops it. Build output (node_modules, target, .build, DerivedData…) is dropped, not archived. Nothing leaves the machine, so no secrets are pushed anywhere.
  • restore: brings the worktree back on its branch (detached if the branch moved away or is checked out elsewhere), with its sparse patterns, files, index and submodules, from the ref or else the bundle.

acpmux:

  • acpmux worktree list|clean|restore|archives (also cmux acp worktree …).
  • The daemon runs a pass when a turn ends, a session closes, or the daemon starts, at most every 6 h / 1 h / 10 min by pressure. It never runs on a timer, runs one pass at a time, and stays off the runtime. Gated by worktrees.autoCleanup (default on; an unreadable settings file turns it off).
  • Restore on reopen: a session that starts again in a reclaimed worktree gets it back before its harness starts (worktree_restored event).
  • _acpmux/worktree/report|clean|restore (local clients only) back the machine view's disk + reclaimable numbers and its Clean up now button (next PR, with the App Store "Auto cleanup" entry from app-store-features).

Relation to existing pieces

  • web/services/vms/hygiene/cmux-worktree-reaper (#22160) and cmux-machine-hygiene (#21795, #22112) are the Cloud-VM shell tools. The reaper holds dirty worktrees, and its opt-in backup pushes wip/ branches. This PR keeps that work in a local archive instead, so the archive needs no secret scan and no approval, and it adds restore and daemon triggers. Every Cloud machine runs this daemon (cmux-tui/AGENTS.md), so it covers them too. Build caches, logs and temp files stay with cmux-machine-hygiene.
  • glaeda-worktree-reclaim (Leo's Macs, Big Red) only handles clean, pushed worktrees. This one subsumes it on machines with cmux.

Testing

  • Unit tests for the policy (sessions, processes, locks, pressure tables, finished, unarchivable), the worktree-list parser, reflog time, lsof parsing, sizes and the store.
  • tests/care_lifecycle.rs runs real repositories end to end: unpushed + staged + deleted + untracked + ignored .env survive reclaim and restore with an identical git status; restore falls back to the bundle after the ref is deleted and gc'd; pushed clean worktrees go without an archive; squash-merged work is detected; nested repos refuse; sparse worktrees come back sparse; a pass archives the idle worktree and keeps the open session's.
  • Hosted: ./scripts/verify-cmux-tui-hosted.sh --filter care_ (all the tests above are named care_* / care_tests::).
    Agent session: cosmic-gray

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Adds automatic git worktree cleanup to machines running cmux so agent worktrees and their build output stop accumulating until the disk fills. The new cmux-worktree-care crate inventories every linked worktree under the default roots, decides per worktree whether it may go, and removes it.

The acpmux daemon runs a pass when a turn ends, a session closes, or the daemon starts — never on a timer — at most every 6 h / 1 h / 10 min depending on disk pressure. Automatic passes run only when the worktrees.autoCleanup setting is true (on by default; an unreadable settings file turns them off); acpmux worktree list|clean|restore|archives exposes the same operations, with manual clean running a pass under the same guards, and _acpmux/worktree/report|clean|restore back the machine view's disk numbers and Clean up now button. A session reopening in a reclaimed worktree gets it back before its harness runs. Sessions come from every resolved acpmux state folder on the machine (BuildIdentity and the worktrees.stateHomes setting); what works in a worktree outside acpmux comes from Claude Code's session registries, never a process scan.

Safety

  • A folder something has open stays: one bounded per-folder lsof check right before removal, and a hit or an error keeps it.
  • Locked worktrees, operations under way, open sessions, and work a registry marks as in use are never touched; a pass re-reads facts and registries right before acting and re-decides, stopping removal on any change. Reclaim also refuses when HEAD moved since the scan; an unborn HEAD is unarchivable.
  • A session registry that reads back no records stops the pass, so a format change cannot silently make every session look gone.
  • Anything used in the last day is left alone — 24 h under 80% disk used, 12 h from 80%, 6 h from 90% — since a shell, build, or editor is only seen through the files it touches.
  • Submodule probes that error or time out fail closed, making the worktree unarchivable; a failed archive ref deletion keeps its manifest so expiry finds it again, and a failed manifest save drops the new ref.
  • A worktree no archive can keep (nested repo, changed submodule, assume-unchanged/skip-worktree files, over 512 MiB) is only trimmed of build output; build output counts only with its marker file.
  • Work that exists only in the worktree becomes git stash-shaped commits under refs/cmux/archive/<id>, plus a bundle and manifest in ~/.cmux/worktree-archive; nothing leaves the machine, and archives expire after 14 days.
  • Worktree admin dirs are pruned only after git gc's own 3 months, so a folder on an unmounted volume or moved by hand keeps its HEAD, index, and reflog; a hung lsof probe is killed after 60 s so it cannot stall all passes.
  • Restore brings the worktree back on its branch (detached if the branch moved) with sparse patterns, files, index, and submodules.

Testing

  • Unit tests for policy rules, the worktree-list parser, reflog time, registry parsing, sizes, and the store.
  • care_lifecycle.rs runs real repositories end to end: unpushed, staged, deleted, untracked, and ignored .env files survive reclaim and restore with an identical git status; bundle fallback, squash-merged detection, and nested-repo and sparse worktree cases are covered.