fix(cloud): retain and retry environment builder cleanup
Failed deletion of an environment builder or warm-up VM currently lets the build settle with no remaining cleanup work, leaving a running VM with idle shutdown disabled. This change will retain cleanup in the provider ledger and retry from CloudDO alarms until deletion or absence is confirmed.
Requested fix: "persist the cleanup work ... and retry it (alarm/backoff) until deletion is confirmed ... Keep idle-shutdown behaviour otherwise unchanged."
Finding 1: https://github.com/manaflow-ai/cmuxterm-hq/issues/4792
Validation is in progress: this first commit is the required failing fake-provider regression. The same focused command will be run before and after the fix, and the results recorded here before handoff. No real provider resources are used.
Risk: cleanup must retain its original route, name and ownership tag and must not delete an active build. Existing guarded deletion remains the deletion boundary.
Observed / Not run
Observed: source confirms both cleanup exceptions are discarded. Focused regression execution pending.
Not run: live provider operations; this change is tested only with fakes.
Agent session: agent session
![View with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/view-with-codesmith-dark-v3-3x.png)
![Autofix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-dark-3x.png)
Need help on this PR? Tag @codesmith-bot with what you need.
Summary by cubic
Fixes environment builder and warm-up VM cleanup so a failed deletion no longer leaves the build settled with a running VM and idle shutdown disabled. Cleanup is now persisted in the ledger as env_cleanup and retried from CloudDO alarms (with backoff) until deletion or absence is confirmed.
Adds a fake-provider regression test covering builder, warm-up, missing, repeated-transient-failure, setup-failure, both, and mismatch cases, asserting retry backoff, alarm persistence, and absence of VMs and pending cleanup after a provider restart. The test serializes and drains alarm execution so backoff and persistence assertions are deterministic. Only the SQLite fake provider is used.
- Atomically hands builder and warm-up names to the cleanup ledger when the build settles.
- Runs cleanup immediately on settlement; backed-off cleanup shares the machine's retry budget without starving its peer.
- Preserves guarded deletion, route, name, and ownership tag; an owner mismatch is rejected with
cloud.provider.name_conflict.