more memory-efficient rollback query
Why
Merge-failure recovery reverses a partial merge by reverting every default-branch
write in the merge window. The existing RollbackQuery ran that revert as one
monolithic Cypher statement — 16 chained per-edge-type phases threading a growing
vertex_ids list — and that shape cannot survive the large databases recovery
exists for:
- It exhausts database transaction memory. Each chained phase gets a planner
Eagerbuffer plus anEagerAggregation, and the buffers stack. Measured on a
22M-edge graph: a 430K-edge rollback blew through the 2.71 GiB default
transaction-memory pool twice before a third attempt fit; a 1.07M-edge rollback
never completed. - It is not crash-safe. The orphan-vertex cleanup and the metadata restore were
driven by vertex ids collected earlier in the same statement. Batches of edge
reversals commit durably as the statement runs, so a crash strands already-reversed
edges whose ids died with the statement — a re-run can no longer find them, leaving
orphaned vertices and merge-time metadata behind while recovery reports success. - Its metadata restore misses new objects. Restoring only vertices that carry a
previous_updated_*snapshot leaves every object that first landed on the default
branch via the failed merge stamped with the merge's timestamp/author (143K vertices
on the 430K-edge test merge).
Goal: make the range rollback memory-bounded, crash-resumable with no state held
outside the database, and metadata-complete — while keeping its callers (merge
failure recovery, mid-merge rollback, schema-update rollback) on the same component.
Part of IFC-2437 (merge failure recovery).
What changed
- Rolling back a large failed merge no longer risks exhausting the database's
transaction memory, and completes in the same ballpark as (measured: faster than)
the previous query. - A recovery interrupted at any point can be re-run to completion: no orphaned
vertices or merge-time metadata are stranded, where previously an interruption
could leave both behind permanently whileRECOVEREDwas still reported. - Objects that were new to the default branch in the failed merge now have their
merge-timeupdated_at/updated_bystamps cleared on rollback (restored to
unset) instead of keeping the failed merge's timestamp.
RollbackQuery is replaced by a GraphRollbacker component (core/rollback.py)
running two statements (core/query/rollback.py):
- Reopen — reset
to/to_user_idon edges closed in the window, one label-less
pass batched withCALL … IN TRANSACTIONS. The metadata restore for each edge's
endpoints rides inside the same batch, so every committed batch is fully finished. - Delete — remove edges created in the window, as two transactional blocks:
the metadata restore first, then the edge deletions with the orphan-vertex cleanup
in the same batch as the deletions that caused it.
Load-bearing decisions, each verified against a 22M-edge dataset:
- Restore-before-delete is the crash-safety mechanism. The planner's
Eager
between the delete statement's two blocks guarantees every restore commits before
the first edge deletion; at any interruption point the not-yet-deleted edges still
match a re-run and completed restores repeat as window-filtered no-ops. The orphan
check cannot be separated from its deletions — an orphan is unfindable once its
edges are gone — so it stays in-batch. - The restore is window-based, not snapshot-driven. Any vertex stamped inside the
window is restored from its snapshot; a vertex stamped without one was new to the
branch, so restoring its NULL snapshot correctly resets it to unset. This removes
the old query'snode_uuidsinput entirely — which an out-of-process
infrahub recovercould never have produced. - Label-less passes instead of per-edge-type phases. The per-type
branch
indexes barely narrow a default-branch rollback (main holds most edges) and the
from/towindow needs a property read per edge regardless: per-type phases
measured 397–437s vs 116–137s for the untyped passes. Composite
(branch, from)/(branch, to)indexes are the upgrade path if rollback
latency ever justifies their merge write amplification.
Measured on the 22M-edge dataset (430K-edge merge, fresh store, cold page cache):
new rollback 55.0s vs old query 45.8s — but the old query leaves 143,356
vertices with merge-time metadata and needs the in-process node_uuids input.
Callers (DiffMerger, MergeFailureRecoverer, SchemaUpdateCoordinator,
branch tasks, CLI) receive GraphRollbacker by constructor injection; the rollback
semantics (RollbackScope, restore_metadata) are unchanged for each.
No graph-schema or index changes, no new dependencies, no API changes. Which writes
a rollback reverses (scope, window, target branch) is untouched.
Impact & rollout
- Backward compatibility: no breaking changes; no data migration. The rollback
is idempotent and resumable, including over state left by an interrupted run of
the previous implementation. - Performance: database transaction memory is now bounded by the batch size
instead of rollback size; application memory is O(1) (the queries return
nothing). Wall time is comparable to the previous query and proportional to
graph size, not merge size. - Config/env changes: none.
- Deployment notes: safe to deploy.
Checklist
- Tests added/updated
- Changelog entry added (
+rollback-metadata-restore.fixed.md) - External docs updated (if user-facing or ops-facing change) — n/a, internal recovery behavior
- Internal .md docs updated (
dev/knowledge/backend/merge-failure-recovery.mdadded and linked frombackend/AGENTS.md,branch-status.md,merge-recompute.md) - I have reviewed AI generated content
Summary by cubic
Replaced the monolithic rollback with a two-phase, batched rollback orchestrated by GraphRollbacker to prevent OOMs and make rollbacks idempotent, resumable, and crash-safe per batch. Metadata restore now runs in-batch and correctly clears merge-time stamps for objects new to the default branch.
- Refactors
- Introduced
GraphRollbackerwith two passes:RollbackReopenEdgesQuery(reopen closed edges + optional endpoint metadata restore) andRollbackDeleteEdgesQuery(optional metadata restore, delete new edges, and remove newly orphaned vertices). - Removed
RollbackQuery; cleanup is bundled with each phase to keep batches atomic and resumable. - Enforced metadata restore only on default/global branches and cleared merge-time stamps for objects first landing on default during a failed merge.
- Updated callers to inject
GraphRollbacker(DiffMerger,SchemaUpdateCoordinator,MergeFailureRecoverer, branch tasks, CLI recover, API schema, dependency builder); adjusted tests. - Used label-less edge scans with
CALL ... IN TRANSACTIONSto reduce memory and avoid planner eager buffers; added merge-failure recovery docs and a changelog entry.
- Introduced