feat(ods): list dev stacks machine-wide, stop orphans by project, and release unused port leases

Merged
DE
3d ago
+2898-66

Description

Adds ods stack ls to list every Onyx dev stack on the machine from any directory, and lets stop/down --project NAME act on a stack whose worktree is gone. The PR also releases port leases when a stack stops owning them.

ods stack ls [--json] [--all] [--orphaned] (internal/devstack/inventory.go, cmd/stack_ls.go)

  • It merges four sources, keyed by Compose project: docker ps -a + docker inspect (labels, state, health), projects/*/owner.json, every reachable state.json (owner state dirs, plus <git-common-dir>/{ods-stack,ods-stack-shared,ods-stack-frontend,worktrees/*/ods-stack,worktrees/*/ods-stack-frontend} from owner, lease, Docker and state roots), and Registry.Leases(). It does not call devstack.Open, and it takes no locks.
  • An Onyx Compose project is one with relational_db + cache services, or one that a state or owner record claims.
  • Statuses: active (an app process is alive, checked with groupIsOurs, so a reused PID does not count), app-stopped (infra runs, phase stopped), infra-only (infra runs, no app process: a Compose-only project or dead processes), stopped, orphaned, and stale-lease (--all only).
  • Orphan rules: the Compose working dir is gone and no state claims the project; a state's root is gone; or the root exists but is no longer a git worktree (state rows, or Docker rows with an owner record whose state is gone). A row counts as orphaned only when something is left to clean up: processes, containers, leases or an owner record.
  • A missing or corrupt state degrades its row (errors). When Docker is down, ls prints a warning and shows rows from saved state only. Exit codes: 0; 3 when --orphaned finds an orphan; 1 when no source can be read.

down/stop --project NAME and prune (internal/devstack/project_down.go)

  • The command resolves the stack through owner.json → state_dir, then any state that claims the project, then Docker labels. It runs from any directory.
  • It takes the state lock (only if the state dir still exists, so git metadata is never recreated) and the project lock. It stops recorded groups with stopServices/stopRecords and runs docker compose -p NAME down from a neutral dir. Volumes are kept.
  • After a successful down it releases the stack's leases and removes owner.json for an orphan.
  • It refuses a stack with a live service under an existing git worktree unless you pass --force. If --project already matches the current checkout's stack, the existing checkout path runs.
  • ods stack prune runs this for every orphaned row after a confirmation (--yes skips it). --volumes has its own confirmation, and --yes does not skip it.

Port lease fixes (ports.go, lifecycle.go, legacy.go)

  • Stack.Stop releases app leases (everything except *_HOST_PORT). Stack.Down releases app leases after stopServices, and every lease after Compose down. New Registry.ReleaseNames.
  • Shared adoption (releaseLegacyLeases, runs after the legacy guard and before leasePorts): each stopped legacy candidate without live process groups loses all its leases, or only its app leases while its containers run.
  • Reclaim: Lease.StateDir is now recorded (LeaseFor). Another owner's lease is reclaimable when its state is stopped or missing, none of its recorded groups is alive, and, for *_HOST_PORT only, its Compose project has no running container. That last check is lazy, so app-port leasing stays fast. Leases without StateDir are reclaimable only for shared/frontend keys, which are their own state dirs.
  • Drift down: choosePort moves a lease to the lowest free port from Preferred when the service is not running. I added PortRequest.Running because Fixed alone is not set for a running service that does not hold its port yet, and moving such a service changed web dev's full-stack owner (TestWebRuntimeUsesTheFullStackWebOwner caught this).

Docs: ods stack help, tools/ods/README.md, tools/ods/integration/README.md, and the onyx-dev-stack skill (a new "Find and stop old stacks" step, and the updated port bullet).

Known limit: git worktree remove deletes <common>/worktrees/<name>/ods-stack, so the recorded PIDs are lost. down --project still removes the containers, leases and owner record, but it cannot find app processes that outlived that removal.

Discovery-first CLI. Bare ods stack prints a short intro, then groups its subcommands with the same AddGroup pattern as the root command: Find stacks (status, ls, logs, plan), Run this worktree's stack (up, start, restart, stop, down), and Clean up (prune). Each ls row carries next, the exact command that stops it, and the text output logs it with log.Info. Empty results and refusals name the command to run.

Example on the VM (an isolated stack, then its worktree deleted with rm -rf):

$ ods stack ls
PROJECT           STATUS       APP  INFRA  WEB  AGE  ROOT
demo-wt-fd863404  app-stopped  -    7/7    -    1m   ~/demo-wt
$ rm -rf ~/demo-wt && ods stack ls --orphaned   # exit 3
PROJECT           STATUS    APP  INFRA  WEB  AGE  ROOT
demo-wt-fd863404  orphaned  -    7/7    -    1m   ~/demo-wt (missing)
level=info msg="To stop demo-wt-fd863404 (orphaned), run: ods stack down --project demo-wt-fd863404"

After stack stop, only the *_HOST_PORT leases remained. After down --project, no containers or leases were left.

How Has This Been Tested?

  • Unit tests for the inventory merge (fake containers, owners, states, leases, liveness): live, app-stopped, infra-only, Compose-only, orphan with a missing root, a removed worktree whose directory remains, a stale lease, Docker unavailable, unreadable sources, and sort order.
  • Real-process tests for DownProject/StopProject: an orphan in a deleted linked worktree is stopped, its leases are released and its owner record is removed; a live worktree is refused without --force.
  • Lease tests: reclaim of a stopped owner, live process group skipped, non-stopped owner skipped, *_HOST_PORT with and without running containers, drift down vs. running, ReleaseNames, stop/down releases, adoption with and without a live legacy group, and the frontend handoff releasing preview leases.
  • Coverage gates: cmd is at 90.6% against a 90.4% floor, and internal/docker at 99.3% against 99.2%.
  • go test ./internal/... ./cmd/ passes, except TestRunCI_failures, which also fails on a clean main in this environment. golangci-lint passes.
  • New integration test TestStackLiveOrphanedWorktree (-tags integration) passed on the VM (246 s). It starts an isolated stack in a disposable worktree, deletes the worktree, asserts that ls --orphaned lists the stack with exit 3, then that down --project stops its processes, removes its containers and releases its leases.

Additional Options

  • [Optional] Please cherry-pick this PR to the latest release version.

Summary by CodeRabbit

  • New Features
    • Added ods stack ls to view active, stopped, orphaned, and stale-lease stacks with service, health, port, age, and location details.
    • Added JSON output, filtering, orphan detection, and project-based shutdown support.
    • Added ods stack prune to safely clean up orphaned stacks, with optional volume removal.
  • Bug Fixes
    • Improved port lease reclamation and reassignment for stopped stacks.
    • stop now releases application ports while preserving infrastructure ports; down releases all ports.
  • Documentation
    • Expanded lifecycle, recovery, listing, and cleanup guidance.