fix(host-service): delete worktrees pinned by a read-only directory

Merged
SA
2mo ago
+303-1

Problem

Deleting a workspace fails, permanently, on a worktree that contains a directory whose owner write bit has been cleared. The destroy saga asks git to remove the worktree; git drops the registration and then fails partway through its own recursive delete with Permission denied. The guarded fallback rm -rf fails the same way, the handler throws, and runDestroy's catch un-archives the row — by design, so the workspace stays visible and retryable rather than orphaning disk. The user retries and gets the identical failure, forever. Nothing in the app can clear it; the only way out is a terminal and chmod -R u+w.

Two machines in HOST-SERVICE-5J show back-to-back retries seconds apart on the same worktree, which is that loop.

Harm: a user who deletes a workspace containing a read-only directory cannot remove it through the product at all, and the failed row keeps coming back into their sidebar. Roughly 6 events across 3 machines in 7 days on 1.27.0.

Reach: the change ships in the next desktop/host-service release (shared version, next after 1.27.0) and reaches every desktop user, since destroy runs in the bundled host-service. The affected code path is the one the reported events execute.

Why code, and not something else: archiving is wrong — the user is genuinely stuck, this is not a telemetry artifact. A Sentry-side filter is repo-law forbidden and would leave the user just as stuck. Deleting the fallback branch is not an option: it exists because git's unregister and its delete are not atomic (#6730). Fixing it at a different layer means telling the user to run chmod themselves, which is strictly worse than doing the one thing that unblocks the delete. No in-flight work removes or restructures this branch (see Neighbours).

Root cause

Removing a directory entry requires write permission on the directory holding it, not on the entry. One read-only directory therefore pins its entire subtree against every remover — git, Node's rm, and the user's own rm -rf.

Reproduced locally against both syscall shapes in the issue:

--- parent dir 0o555, child dir empty-able
  code=EACCES syscall=rmdir path=<tree>/evals/results
--- dir itself 0o555 with children
  code=EACCES syscall=unlink path=<tree>/evals/results/out.json
--- dir 0o000
  code=EACCES syscall=scandir path=<tree>/evals/results

Those match the two distinct sampled events exactly — one rmdir '<worktree>/evals/results' (a runner's output folder), one unlink '<worktree>/pkg/mod/gopkg.in/yaml.v3@v3.0.1/README.md'. The second names the cause outright: go mod download marks every directory in the module cache 0555, which is why go clean -modcache exists. This is normal tooling output, not corruption, so the condition will keep recurring.

Note the failing path in the unlink shape is a file — chmod'ing it changes nothing; the directory above it is what denies. That rules out repairing the single path named in the error.

Fix

New remove-directory-tree.ts, called from the one throw site in the events:

try {
  await rm(path, { recursive: true, force: true });
  return;
} catch (error) {
  if (!isPermissionDenied(error)) throw error;
}
await restoreOwnerDirectoryAccess(path);
await rm(path, { recursive: true, force: true });
  • One retry, not a loop. The recovery pass is exhaustive over the tree, so a second denial is something chmod cannot fix and must be reported.
  • A full pass, not a targeted chmod. Measured: a per-error-path chmod needs one retry per read-only directory — 4 retries for a 3-directory fixture, and a module cache has thousands. A 1001-read-only-directory tree is cleared by the full pass plus one retry in 149 ms.
  • Directories only, symlinks never followed. A link's dirent reports as a link, so the pass cannot leave the tree. It only ever adds owner bits (mode | 0o700), never widens group or other.
  • Blast radius is the existing gate. The pass is rooted at local.worktreePath, which isInsideProjectWorktreesRoot has already cleared; no second, weaker check was added.
  • No behaviour change for anything else. The thrown TRPCError, its message, the un-archive, and the isMissingPath disk-state recheck that reports the outcome are all untouched.
  • No typed cause. Nothing reads one here — no renderer branch, no retry decision, no test — so the throw stays a plain TRPCError({ code, message }).

What the matcher must not match

isPermissionDenied is code === "EACCES" and nothing else. It is checked off the errno, not the prototype, per the worker-boundary rule; it also cannot key off the message, because Bun reports the failing syscall as rm where Node reports rmdir/unlink.

Every one of these shares this exact Sentry fingerprint and must keep failing as it does today:

errnowhy it must not be rescued
ENOTEMPTYa live writer re-creating files under the delete — 5+ events in this same issue
EPERMimmutable flag or foreign ownership; a chmod by this process cannot lift it either
EBUSYnot a permission failure
Error with no code, a string, nullnot an fs error at all

ENOTEMPTY is the one that matters: it is in this fingerprint right now, and silencing it would hide a real bug.

Verification

The new test reproduces the issue on unmodified main and passes after (test/workspace-cleanup.test.ts):

TRPCError: Worktree at <base>/p-1/wt-readonly is no longer registered with git,
but its folder could not be removed: EACCES: permission denied, rm '<base>/p-1/wt-readonly'
(git worktree remove: error: failed to delete 'wt-readonly': Permission denied)
  at runDestroyPhases (.../workspace-cleanup.ts:582:13)
(fail) ... destroy restores write permission and removes it [2.53ms]
 32 pass / 1 fail

The negative tests were written first and pass both before and after — they are regression guards on behaviour that must not move. Tests added: a read-only evals/results and a module-cache-shaped subtree are removed end-to-end through the router; a permission denial above the worktree still throws and leaves the folder, with the containing directory's mode asserted unchanged (the pass does not climb out); the errno table above; a non-permission failure (ENOTDIR) rethrown untouched; a symlink to an outside read-only directory left with both its contents and its mode intact.

$ bunx biome check <4 changed files>
Checked 4 files in 15ms. No fixes applied.

$ cd packages/host-service && bun run typecheck
$ tsc --noEmit --emitDeclarationOnly false      # exit 0, no diagnostics

$ cd packages/host-service && bun test
 1696 pass
 8 todo
 0 fail
 3967 expect() calls
Ran 1704 tests across 165 files. [222.49s]

Production runs the host-service under Node, but the suite runs under Bun, and the two have different fs.rm implementations — so the shipped module was also exercised directly under Node:

node v26.8.1: read-only tree removed = true
node v26.8.1: isPermissionDenied EACCES=true ENOTEMPTY=false

No desktop git code was touched, so apps/desktop/src/no-main-process-blocking.test.ts does not apply. Repo-wide bun run lint was not run (hangs on cross-worktree bun locks).

Neighbours

Based on origin/main. Both open PRs against this file were read:

  • #7094 fix/worktree-delete-parallel-rm — its only hunk in workspace-cleanup.ts is at the cleanupGitOps.removeWorktree call (~L526-552); mine is at ~L579. No textual conflict. Semantically it does interact: it moves the native rm into gitWorktreeRemoveTask before git unregisters, and that rm is unguarded, so once it lands this failure would surface earlier as Failed to verify worktree removal and never reach the branch fixed here. The follow-up is one import — have that task call removeDirectoryTree — but it is not built here, since #7094 is unmerged and guessing at its final shape would be speculative.
  • #7209 archive-delete-experience — restructures tombstone bookkeeping and rewrites comments in runDestroyPhases, but its hunks are at L2-23, L153-227, L248-276, L347-362, L382-389 and L679-687. None touch L550-600. The only expected conflict is trivial: we both add to the import block at the top of the file. Neither PR touches test/workspace-cleanup.test.ts.

Adjacent throw sites, left alone

Two other rm(local.worktreePath, …) call sites in the same function have the same latent exposure and are deliberately unchanged, since neither appears in the reported events: the session-folder removal (~L455, gated by isInsideSessionsRoot) and the project-repo-missing removal (~L500). The session branch does appear in this Sentry fingerprint, but with ENOTEMPTY, which is a different cause.

Note HOST-SERVICE-5J is a mixed group — its 27 events span at least five root causes (this one, ENOTEMPTY live-writer races, git validation failed, and ~10 git/removeWorktree 120 s timeouts, which are #6887 / #7094's territory). This PR fixes the permission slice only, so the issue will keep receiving events; it should not be resolved on this merge.

Refs HOST-SERVICE-5J

https://claude.ai/code/session_0183p4FE6szo6NcATmp1kvwY


Summary by cubic

Fixes workspace deletion failing permanently when a worktree contains a read-only directory. The destroy flow now restores owner write access on blocked directories and retries the removal once.

Bug Fixes

  • Replaces the plain rm fallback with removeDirectoryTree, which recovers only from EACCES and rethrows ENOTEMPTY, EPERM, and EBUSY untouched.
  • The recovery pass chmods directories only, never follows symlinks, and stays within the already-gated local.worktreePath.
  • Adds tests for read-only evals/results and Go module cache-shaped subtrees, a permission denial above the worktree, and symlink escapes.
  • Only the permission slice is fixed; HOST-SERVICE-5J will keep receiving events for other root causes and should not be resolved on this merge.

Summary by CodeRabbit

  • Bug Fixes
    • Workspace cleanup now successfully removes worktrees containing read-only directories by restoring required owner permissions and retrying deletion.
    • Cleanup preserves existing error behavior for failures unrelated to permissions.
    • Permission problems outside the worktree remain reported without altering the affected project directory.