PR #6771 - superset-sh/superset

Problem

Deleting a workspace runs git worktree remove as a worker-pool task with a
two-minute budget. On some machines the task never returns, the pool kills it,
the deletion fails, and the user retries into the same two-minute wait. The
error that reaches Sentry says only:

Failed to verify worktree removal at <path>: [host-worker] Task "git/removeWorktree" timed out after 120000ms

That is the whole report. The task does three things that stall for entirely
unrelated reasons — a blocking realpathSync.native on the worktree path, the
git worktree remove --force --force subprocess (which recursively deletes the
tree), and the git worktree list --porcelain read afterwards — and nothing
distinguishes them. The condition has been raised as needing an owner on four
consecutive triage runs and has never been diagnosable from its own telemetry.

User-visible harm: a user deleting a workspace waits two minutes, gets a
failure, and the workspace is still there; retrying costs another two minutes.
HOST-SERVICE-17 has 192 occurrences between 2026-08-10 and today on one machine
(release 1.23.0); HOST-SERVICE-47 is the same condition on a second machine in a
different organization (6 occurrences, release 1.24.1). Both are live released
builds.

This PR does not fix that hang — we still do not know what hangs, which is
the point. It makes the next occurrence say which step was in flight.

Will this reach the reported failure? Yes, and the path is exact. The Sentry
event is the TRPCError thrown at workspace-cleanup.ts:525, and its message
interpolates the pool's timeout text verbatim (it is also the trpc_message
extra). Enriching that text puts the phase in the issue title and in the event
body. The captured events carry no (inline) suffix, so they came from the
worker path — which is what this instruments; the inline fallback is covered
too. Sentry groups these by stacktrace, not message (192 events with 192
different worktree paths share one group), so the existing issues keep
collecting rather than fragmenting.

Why code, versus the alternatives? Archiving is what already happened to
HOST-SERVICE-47 (archived_until_escalating) and the condition simply came back
on another machine. The code path cannot be deleted — it is the core of workspace
deletion. An inbound Sentry filter is forbidden by repo law (#6164) and would
destroy the only signal there is. Fixing at another layer — raising the budget,
retrying, making removal more tolerant — would mask the hang without explaining
it. Doing nothing is what four triage runs already did. The missing fact exists
only inside the worker at the moment it stalls, and nothing but the worker can
observe it.

No classifier or matcher is added here.

Root cause of the blind spot

The budget is enforced by a setTimeout in the parent (WorkerTaskRunner),
while the phases run inside the worker thread. When the timer fires, the parent
knows the elapsed time and nothing else — and it immediately retires the worker,
so there is nothing left to interrogate afterwards either.

Design choice (worker reports outward, not parent reconstructs). The parent
has no way to reconstruct the phase: it holds one timer and one opaque task
payload, the handler's progress leaves no trace it can read, and the worker is
killed the instant the budget expires. The only shape that reconstructs anything
in the parent would be splitting git/removeWorktree into one pool call per
step, which would change the deletion's failure semantics — a partial removal
would become an observable intermediate state the saga has to handle — and that
is out of scope for a capture-enrichment change. So the worker announces each
step as it starts, the parent keeps the last announcement, and the timeout names
it. That is one message kind, one optional callback, and one string; there is no
progress framework here, no percentages, no subscribers, and a task that reports
nothing produces byte-identical output to before.

Fix

worker-task-protocol.ts: a {kind: "phase", taskId, phase} message and its
type guard.
define-worker-task.ts: handlers get an optional second argument,
reportPhase(phase). Existing handlers are untouched and ignore it.
host-worker.ts: binds reportPhase to the running task's id.
WorkerTaskRunner.ts: records the last phase per task; the timeout error
appends in phase "<phase>" when one was reported, leaving the message
unchanged otherwise.
host-worker-pool.ts: same for the inline fallback, which enforces the same
budget and would otherwise stay undiagnosable.
tasks/git.ts: gitWorktreeRemoveTask reports resolve-path,
worktree-remove, worktree-list. The first statement is labelled so no
moment of the handler is unlabelled — an unlabelled timeout would be
indistinguishable from one produced by a build without this change.

Nothing else changes: not the 120000ms budget, not retry behaviour, not what the
user sees, not the deletion logic.

A note on the brief: it described the middle step as "delete a leftover
directory directly". That step exists but lives in runDestroyPhases in the
parent process, outside the worker budget, so it cannot be the step that
expires. The handler's actual three steps are the ones labelled above.

Verification

All run on this branch; output pasted verbatim.

bunx biome check on the changed files:

Checked 8 files in 10ms. No fixes applied.

cd packages/host-service && bun run typecheck:

$ tsc --noEmit --emitDeclarationOnly false
TYPECHECK EXIT=0

bun test packages/host-service — the whole package, including
test/workspace-cleanup.test.ts, which #6753 also edits:

 1213 pass
 8 todo
 0 fail
Ran 1221 tests across 115 files. [213.91s]

Proof of the outcome, not an assertion of it

Four tests in host-worker-pool.test.ts shim a git onto the front of PATH
that blocks on a chosen subcommand, then drive the real
gitWorktreeRemoveTask through the real pool, worker thread and task
registry — the only fakes are git itself and the budget — and assert on the
exact message the caller receives:

(pass) HostWorkerPool > worker timeout names the phase that was running (worktree-remove)
(pass) HostWorkerPool > worker timeout names the phase that was running (worktree-list)
(pass) HostWorkerPool > a task that reports no phase keeps its timeout message unchanged
(pass) HostWorkerPool > inline timeout names the phase too

The two removeWorktree cases are the same task with the same budget and the
same shim, differing only in which subcommand stalls — so a label that tracked
the task rather than the step, or one left over from the step before, fails one
of them. The third is the other half of the contract: git/worktreeState
reports no phases, so its timeout must read exactly as it did before.

With the three reportPhase calls disabled, the three phase tests fail and the
no-phase test still passes — they are not vacuous, and the no-phase test is not
secretly coupled to the machinery:

(fail) HostWorkerPool > worker timeout names the phase that was running (worktree-remove)
(fail) HostWorkerPool > worker timeout names the phase that was running (worktree-list)
(fail) HostWorkerPool > inline timeout names the phase too
 11 pass
 3 fail

Two determinism notes, because a timing-dependent test that hangs a subprocess
is easy to get subtly wrong:

The worker is warmed before each timed run. A cold worker compiles the
whole task module graph first — measured at ~305ms against a 1000ms budget on
an idle machine — which could burn the budget before the handler ran at all,
timing the task out having reported nothing. Warm, the first phase is reported
within a message hop of dispatch, so the margin stops depending on machine
load.
The shim blocks on a sentinel, not a sleep. A fixed sleep long enough
to survive a loaded machine is also one left running after the test, and one
short enough to clean up can expire mid-assertion and let the task succeed —
a false pass. The shim instead spins until its own file is removed, which
afterEach's existing fixture cleanup does, giving an unbounded hang with
bounded cleanup.

The suite passes 3/3 under 16-way CPU contention as well as idle.

A fourth test in test/workspace-cleanup.test.ts closes the last link — that the
label survives into the error the reporter actually captures. It is a unit test
of the interpolation: it builds its expectation from the injected pool error
rather than restating it, so the two halves cannot drift, and the pool's exact
wording is pinned separately by the tests above.

Repo-wide bun run lint was not run — it hangs on cross-worktree bun locks.

Not covered by a test: the resolve-path label. Making realpathSync.native
block on demand is not something a test can arrange portably, so that one line
rests on inspection.

Stacked on #6753 (fix/delete-orphan-worktree-fallback), not main — that PR
edits workers/tasks/git.ts and test/workspace-cleanup.test.ts too, and this
builds on top of it without reverting or restating any of it. Review the single
commit d4b037b9a; everything below it belongs to #6753.

Note for whoever merges: #6753 is a fork PR, so its branch did not exist on
superset-sh/superset and could not be used as a base ref. The base branch here
is a mirror of #6753's head (95c5d2112) pushed to this repo purely to make the
stack expressible — delete fix/delete-orphan-worktree-fallback once #6753
merges, and retarget this PR at main.

Both issues stay open as their own trackers. This does not resolve the hang;
it makes the next occurrence say where it hung — hence Refs, not Fixes.

Refs HOST-SERVICE-47
Refs HOST-SERVICE-17

https://claude.ai/code/session_01BrNKdKTC54LnA9jdpDDtA8

Summary by cubic

Names the in-flight phase in worker task timeout errors so triage can see which step hung. Previously the timeout named only elapsed time; now it appends the last reported phase, improving signal for git/removeWorktree timeouts (HOST-SERVICE-17, -47).

Adds a worker→parent phase message and guard; WorkerTaskRunner records the last phase and appends it to timeout errors; the inline path does the same.
Extends define-worker-task so handlers can accept an optional reportPhase(phase); existing handlers need no changes.
Instruments gitWorktreeRemoveTask to report resolve-path, worktree-remove, and worktree-list.
Keeps all other behavior unchanged: budgets, retries, deletion logic, and Sentry grouping. Tasks that report no phase keep the exact prior timeout message.

Summary by CodeRabbit

New Features
- Added progress phase reporting for background tasks.
- Timeout errors now identify the task phase that was active when the timeout occurred.
- Git worktree cleanup reports progress during path resolution, removal, and verification.
Bug Fixes
- Improved timeout diagnostics for both worker-based and inline task execution.
Tests
- Added coverage for phase-aware timeout messages and cleanup error propagation.