merge failure recovery improvements
Why
Live failure-mode testing of the merge-failure-recovery feature surfaced two ways a crashed branch merge leaves the system in a degraded state that infrahub recover merge did not clean up:
- stale destination schema metadata. When a merge crashes after the schema graph is written, recovery rolls the graph back but left the destination branch's persisted
schema_hash/schema_changed_atat their failed-merge values. Those are plainBranch-vertex properties written in place mid-merge, so the versioned range rollback cannot revert them. The persisted metadata then disagrees with the rolled-back graph, so every worker keeps detecting a "new" schema hash and churning through reconciles. - leaked distributed locks. A hard-killed merge leaks no-TTL locks (
lock.global.graph,lock.global.schema, and the per-branchlock.diff-update.*locks). Recovery cleared only the merge lock, so the leaked locks silently blocked the next merge until the 15-minute deadlock reaper cleared them — with no log line before the blocking acquire to diagnose the hang.
Goal: make infrahub recover merge leave the destination branch's schema metadata and the distributed locks in the same state a healthy merge would, and make a leaked-lock hang diagnosable from logs.
Non-goals: no change to the merge happy path, the rollback range logic, or the failure-detection/MERGE_FAILED flagging.
What changed
Behavioral changes
- Recovery now realigns the destination branch's
schema_hash(recomputed from the rolled-back graph) andschema_changed_at(restored from the value captured at merge start), so a recovered merge no longer triggers spurious "New hash detected" reconcile churn. - Recovery now clears the diff-update / global-graph / global-schema locks a dead merge leaked, deleting only those whose holder is no longer an active worker — a lock a live flow still holds is left in place — so the next merge is not silently blocked until the deadlock reaper runs.
- The merge path now logs before the blocking merge-lock and global-graph-lock acquisitions, so a leaked-lock hang is diagnosable from the logs.
Implementation notes
- New
Branch.pre_merge_destination_schema_changed_at(optional, defaultNone) is stamped onto the source branch at merge start (extracted intoBranchMergeOrchestrator._record_merge_start) and is the source recovery restores from. - New
StaleLockCleanercomponent (liveness-guarded deletes) and the shared active-worker lookup consolidated ontoInfrahubComponent.list_active_worker_ids. - The recovery schema load is isolated behind a single override point (
MergeFailureRecoverer._load_destination_schema) so tests can substitute an in-memory schema and skip the slow database round-trip.
Impact & rollout
- Backward compatibility: new optional
Branchfield, no migration; existing in-flight/olderMERGE_FAILEDrecords without the field are handled (schema hash still recomputed,schema_changed_atleft untouched). - Performance: no runtime impact; the schema-load override is a test-only seam.
- Config/env changes: none.
- Deployment notes: safe to deploy.
Checklist
- Tests added/updated
- Changelog entry — n/a, the merge-failure-recovery feature is still unreleased
- External docs updated (if user-facing or ops-facing change)
- Internal .md docs updated (internal knowledge and AI code tools knowledge)
- I have reviewed AI generated content
Summary by cubic
Improves merge failure recovery by restoring destination schema metadata and clearing stale distributed locks so subsequent merges and workers don’t get stuck or recompute endlessly. Adds merge-start bookkeeping and better lock diagnostics for easier debugging.
-
New Features
- Capture the destination branch’s
schema_changed_atat merge start on the source branch (newBranchfield:pre_merge_destination_schema_changed_at). - Clear stale diff-update and global graph/schema locks during recovery using a new
StaleLockCleaner(deletes only locks held by dead workers). - Introduce
WorkerLivenessprotocol and expose globalInfrahubComponent.list_active_worker_ids; log before/after acquiring global merge/graph locks; CLI recovery wires inSchemaManager,StaleLockCleaner, andDiffLocker.
- Capture the destination branch’s
-
Bug Fixes
- Recovery realigns the destination schema metadata with the rolled-back graph: recomputes schema hash and restores
schema_changed_atfrom the captured pre-merge value, including when it wasNone. - Recovery sequence is idempotent and failure-safe: rollback → restore schema metadata → reset proposed change → release merge lock → clear stale locks → reset branch → lift write protection.
- Recovery realigns the destination schema metadata with the rolled-back graph: recomputes schema hash and restores