fix(cloud): retain and retry environment builder cleanup

Closed
TE
7d ago
+118-7

Failed deletion of an environment builder or warm-up VM currently lets the build settle with no remaining cleanup work, leaving a running VM with idle shutdown disabled. This change will retain cleanup in the provider ledger and retry from CloudDO alarms until deletion or absence is confirmed.

Requested fix: "persist the cleanup work ... and retry it (alarm/backoff) until deletion is confirmed ... Keep idle-shutdown behaviour otherwise unchanged."

Finding 1: https://github.com/manaflow-ai/cmuxterm-hq/issues/4792

Validation is in progress: this first commit is the required failing fake-provider regression. The same focused command will be run before and after the fix, and the results recorded here before handoff. No real provider resources are used.

Risk: cleanup must retain its original route, name and ownership tag and must not delete an active build. Existing guarded deletion remains the deletion boundary.

Observed / Not run

Observed: source confirms both cleanup exceptions are discarded. Focused regression execution pending.
Not run: live provider operations; this change is tested only with fakes.

Agent session: agent session


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes environment builder and warm-up VM cleanup so a failed deletion no longer leaves the build settled with a running VM and idle shutdown disabled. Cleanup is now persisted in the ledger as env_cleanup and retried from CloudDO alarms (with backoff) until deletion or absence is confirmed.

Adds a fake-provider regression test covering builder, warm-up, missing, repeated-transient-failure, setup-failure, both, and mismatch cases, asserting retry backoff, alarm persistence, and absence of VMs and pending cleanup after a provider restart. The test serializes and drains alarm execution so backoff and persistence assertions are deterministic. Only the SQLite fake provider is used.

  • Atomically hands builder and warm-up names to the cleanup ledger when the build settles.
  • Runs cleanup immediately on settlement; backed-off cleanup shares the machine's retry budget without starving its peer.
  • Preserves guarded deletion, route, name, and ownership tag; an owner mismatch is rejected with cloud.provider.name_conflict.