fix(host-service): make workspace delete reliably tear down terminal sessions

Merged
SA
5mo ago
+142-9

Problem

A host with automated stale-workspace deletion accumulated ~400 live terminal sessions across already-deleted workspaces. Reported as "delete just removes the git worktree and leaves the sessions running."

This turned out to be a reliability bug, not a missing feature. The cascade already exists: CLI delete_workspace → host-service workspace.delete → destroyWorkspace, whose Step 2a calls disposeSessionsByWorkspaceId before removing the worktree. The problem is that the cascade could silently fail and orphan the PTY:

  1. disposeSessionAndWait marked the sqlite row disposed before confirming the daemon actually killed the PTY. A transient daemon failure left a live process behind a disposed row — invisible to every future workspace-scoped cleanup.
  2. The pty-daemon has no workspace mapping (SessionInfo is just {id,pid,cols,rows,alive}). The terminal→workspace link lives only in sqlite origin_workspace_id, and the onDelete: "set null" FK severs it once the workspace row is deleted — so an already-orphaned PTY can't be found by workspace at all.

Worktree removal reads git's durable on-disk registry so it always succeeds; terminal disposal trusted a mutable sqlite table and optimistically marked rows disposed — hence the asymmetry.

Changes

  • fix: mark disposed only after confirmed kill — failed kills now stay active and retryable/reapable instead of being lost behind a disposed row.
  • fix: clear dead terminal rows when destroying a workspace — the destroy saga deletes the workspace's confirmed-dead terminal rows (status ≠ active), so its session index dies with it instead of lingering as set null orphans. Failed kills (active) are deliberately kept reachable for the reaper.
  • feat: reap orphaned daemon PTYs — a reaper lists the daemon's live sessions and kills any whose row is disposed/exited/null-workspace (or row-less for two consecutive passes, so a being-born session — daemon PTY opened before the sqlite row insert — is never killed mid-creation). Runs at startup (drains the pre-existing ~400 on next host-service restart) and every 5 min; interval is unref()'d and cleared in dispose().

Why not onDelete: cascade?

An FK action fires at the DB level and can't know whether the PTY was actually killed. cascade would delete the row for a still-alive failed-kill PTY, turning it into a row-less orphan that's ambiguous with a being-born session (slower two-pass reap). set null leaves an unambiguous active+null row the reaper catches in one pass. Row lifecycle is therefore application-driven and conditional on a confirmed kill; the FK stays set null as a dumb integrity backstop.

Notes for deploy

  • The existing ~400 orphans drain on the next host-service restart (boot reaper) or within one 5-min tick after deploy.
  • Dev mode kills the daemon on exit, so this matters for production/detached daemons — the affected setup.

Verification

  • bun run typecheck ✓
  • bun run lint exit 0 ✓
  • 352 host-service unit tests pass (bun 1.3.11)
Open in Stage

Summary by cubic

Ensure workspace deletion reliably tears down terminal sessions by fixing disposal semantics and adding a background reaper in host-service. The reaper now starts from serve after daemon bootstrap, avoids overlapping passes, and drains existing orphans on restart or within 5 minutes.

  • Bug Fixes

    • Mark a session disposed only after the daemon kill succeeds; failed kills stay active and retryable.
    • Reaper robustness: start from serve to avoid early daemon connections; skip overlapping passes, keep pending state per process, retry failed second‑pass rowless kills; include workspace id in the terminal-row cleanup warning.
    • On workspace destroy, delete that workspace’s non-active terminal rows to avoid orphaned records.

  • New Features

    • Add a terminal reaper that lists live daemon sessions and kills orphans (disposed/exited/null-workspace, or row-less on two passes). Runs on boot and every 5 minutes.

Summary by CodeRabbit

  • New Features

    • Background reaper now runs automatically to detect and clean orphaned terminal sessions.

  • Bug Fixes

    • Terminal sessions are marked disposed only when closure succeeds.
    • Workspace destroy now clears non-active terminal session records during local cleanup.

  • Chores

    • Improved error handling and non-blocking logging for cleanup operations.