infrahub recover merge CLI tool

Merged
AJ
4mo ago
+1689-261

Why

The detection PR flags a merge whose worker died mid-flight as MERGE_FAILED and holds the write block, and the rollback-core PR (#9878) gave us an idempotent, state-free range rollback keyed on (default branch, merge_started_at). But there was still no way to actually use any of it: a failed merge left the default branch write-blocked and the branch stuck, with no operator-facing path back to a writable state.

Goal: the operator-facing recovery entry point — an infrahub recover merge CLI backed by a MergeFailureRecoverer — that reverses the partial merge, resets the branch and any associated proposed change to OPEN, releases the stale merge lock, and lifts the write protection so the default branch is writable again. Idempotent and operator-confirmed.

Why it's safe: recovery reuses the range rollback from #9878, which is safe precisely because the write block owned every default-branch write in the merge window. The recoverer only orders the surrounding state transitions (proposed change, branch status, merge lock, write protection) so that any mid-sequence failure leaves the branch flagged and protected for a clean re-run.

Non-goals:

  • the range rollback query itself — landed in #9878.
  • the recurring detection scan that flags MERGE_FAILED — prior detection PR.
  • other recovery modes (stuck lock, failed branch delete)
    This closes IFC-2566 (reset the app to its pre-merge state, operator-facing).

Closes IFC-2566

What changed

Behavioral changes (what you'll observe differently):

  • New CLI command infrahub recover merge [branch]. With no argument it auto-detects the failed merge; given a branch name it targets that branch. It previews first and asks for confirmation (--yes/-y to skip). By default it only recovers a merge whose worker is confirmed dead; --force/-f also recovers a merge stuck with an absent/ambiguous lock. Exits non-zero when recovery fails.
  • A branch left in MERGE_FAILED can no longer be deleted until it has been recovered — BranchDelete raises MergeRecoveryRequiredError rather than deleting a half-merged branch.
  • Recovery releases the stale global merge lock the dead worker never freed, so the next merge is not blocked until the deadlock-cleanup cron runs.

Internals:

  • MergeFailureRecoverer exposes preview() (read-only: reports RECOVERABLE/NOTHING_TO_RECOVER, makes no changes) and recover(). Outcomes: NOTHING_TO_RECOVER, RECOVERABLE, RECOVERED, ORPHANED_CLEARED, FAILED.
  • Recovery sequence, all inside one try/except so a failure at any step returns FAILED and leaves the branch flagged + protected for a re-run: rollback → reset proposed change → release merge lock → reset branch to OPEN → lift write protection. The proposed change is reset before the branch, and the write protection lifted last, so a partial run never reopens the branch or unblocks writes prematurely. The merge lock is released while protection is still up (so this unconditional delete can only hit the dead worker's lock, never a fresh merge's), and a release failure surfaces as FAILED rather than being swallowed — a held lock would otherwise block every future merge silently.
  • Stale-key handling: when nothing needs recovering, recover() clears an orphaned write-protection key whose branch was removed out-of-band, is malformed, or is already OPEN (ORPHANED_CLEARED), but leaves a key naming a healthy in-progress MERGING branch in place. A transient/unreadable cache value is never treated as absent.
  • MergeFailureIdentifier reshaped so its dead-merge decision is reusable by recovery: should_mark_as_failed_merge(branch, now) and merge_lock_holder() are now async instance methods that read the cache themselves, and default_branch is constructor-injected. RecoveryOutcome/RecoveryReport moved out of the identifier into the recoverer.
  • The CLI builds a registry-independent SchemaManager (internal + core models) and attaches it to the db, so the proposed-change lookup during recovery resolves its schema without depending on — or mutating — the global registry or any user-defined schema.

What stayed the same: the healthy-merge happy path; the range rollback semantics from #9878; no database migration, schema, or API-contract change.

Suggested review order

  1. backend/infrahub/core/merge/failure_recoverer.py — preview/recover, the recovery step ordering and failure semantics, the merge-lock release, and orphaned/stale-key clearing.
  2. backend/infrahub/cli/recover.py — the CLI: preview→confirm→recover flow, exit codes, registry-independent schema wiring.
  3. backend/infrahub/core/merge/failure_identifier.py — the async should_mark_as_failed_merge / merge_lock_holder refactor and the moved report types.
  4. backend/infrahub/graphql/mutations/branch.py — the MERGE_FAILED delete guard.
  5. Tests — test_recovery.py (outcomes, force gating, stale-key clearing, delete gate, merge-lock release + retry), test_recovery_rollback.py (end-to-end rollback through recovery), test_failure_identifier.py (replaces test_failed_merge_predicate.py), and integration_docker/test_merge_kill_recovery.py (kill a merge mid-flight, then recover).

Impact & rollout

  • Backward compatibility: additive — a new CLI command plus a delete guard on MERGE_FAILED branches. No schema, API, enum, or config change; no migration.
  • Dependencies: builds on the range rollback (#9878, merged) and the detection PR.
  • Deployment notes: safe to deploy; the new behavior only manifests when a merge has failed.

Checklist

  • Tests added/updated
  • Changelog entry — changelog/+merge-recover-command.added.md
  • External docs — n/a (operator CLI documented via the command's own --help)
  • Internal .md docs updated
  • I have reviewed AI generated content

Summary by cubic

Adds a new infrahub recover merge CLI to roll back partial default-branch writes, reset the failed branch and any linked proposed change to OPEN, and restore write access by clearing protections and any stale merge lock. MERGE_FAILED branches are read-only and undeletable until recovery runs, closing IFC-2566.

  • New Features

    • infrahub recover merge [branch]: previews, asks for confirmation (--yes to skip), and is idempotent.
    • Auto-detects a failed merge or targets a branch; --force can recover when the lock state is ambiguous.
    • Uses the branch’s previewed snapshot for rollback when available.
    • Deleting a MERGE_FAILED branch raises MergeRecoveryRequiredError with MERGE_RECOVERY_REQUIRED_MESSAGE.
  • Bug Fixes

    • Stop advancing a branch’s branched_from after merge.
    • Clear any leftover global merge lock during recovery.