more memory-efficient rollback query

Merged
AJ
4mo ago
+420-175

Why

Merge-failure recovery reverses a partial merge by reverting every default-branch
write in the merge window. The existing RollbackQuery ran that revert as one
monolithic Cypher statement — 16 chained per-edge-type phases threading a growing
vertex_ids list — and that shape cannot survive the large databases recovery
exists for:

  • It exhausts database transaction memory. Each chained phase gets a planner
    Eager buffer plus an EagerAggregation, and the buffers stack. Measured on a
    22M-edge graph: a 430K-edge rollback blew through the 2.71 GiB default
    transaction-memory pool twice before a third attempt fit; a 1.07M-edge rollback
    never completed.
  • It is not crash-safe. The orphan-vertex cleanup and the metadata restore were
    driven by vertex ids collected earlier in the same statement. Batches of edge
    reversals commit durably as the statement runs, so a crash strands already-reversed
    edges whose ids died with the statement — a re-run can no longer find them, leaving
    orphaned vertices and merge-time metadata behind while recovery reports success.
  • Its metadata restore misses new objects. Restoring only vertices that carry a
    previous_updated_* snapshot leaves every object that first landed on the default
    branch via the failed merge stamped with the merge's timestamp/author (143K vertices
    on the 430K-edge test merge).

Goal: make the range rollback memory-bounded, crash-resumable with no state held
outside the database, and metadata-complete — while keeping its callers (merge
failure recovery, mid-merge rollback, schema-update rollback) on the same component.

Part of IFC-2437 (merge failure recovery).

What changed

  • Rolling back a large failed merge no longer risks exhausting the database's
    transaction memory, and completes in the same ballpark as (measured: faster than)
    the previous query.
  • A recovery interrupted at any point can be re-run to completion: no orphaned
    vertices or merge-time metadata are stranded, where previously an interruption
    could leave both behind permanently while RECOVERED was still reported.
  • Objects that were new to the default branch in the failed merge now have their
    merge-time updated_at/updated_by stamps cleared on rollback (restored to
    unset) instead of keeping the failed merge's timestamp.

RollbackQuery is replaced by a GraphRollbacker component (core/rollback.py)
running two statements (core/query/rollback.py):

  1. Reopen — reset to/to_user_id on edges closed in the window, one label-less
    pass batched with CALL … IN TRANSACTIONS. The metadata restore for each edge's
    endpoints rides inside the same batch, so every committed batch is fully finished.
  2. Delete — remove edges created in the window, as two transactional blocks:
    the metadata restore first, then the edge deletions with the orphan-vertex cleanup
    in the same batch as the deletions that caused it.

Load-bearing decisions, each verified against a 22M-edge dataset:

  • Restore-before-delete is the crash-safety mechanism. The planner's Eager
    between the delete statement's two blocks guarantees every restore commits before
    the first edge deletion; at any interruption point the not-yet-deleted edges still
    match a re-run and completed restores repeat as window-filtered no-ops. The orphan
    check cannot be separated from its deletions — an orphan is unfindable once its
    edges are gone — so it stays in-batch.
  • The restore is window-based, not snapshot-driven. Any vertex stamped inside the
    window is restored from its snapshot; a vertex stamped without one was new to the
    branch, so restoring its NULL snapshot correctly resets it to unset. This removes
    the old query's node_uuids input entirely — which an out-of-process
    infrahub recover could never have produced.
  • Label-less passes instead of per-edge-type phases. The per-type branch
    indexes barely narrow a default-branch rollback (main holds most edges) and the
    from/to window needs a property read per edge regardless: per-type phases
    measured 397–437s vs 116–137s for the untyped passes. Composite
    (branch, from)/(branch, to) indexes are the upgrade path if rollback
    latency ever justifies their merge write amplification.

Measured on the 22M-edge dataset (430K-edge merge, fresh store, cold page cache):
new rollback 55.0s vs old query 45.8s — but the old query leaves 143,356
vertices with merge-time metadata and needs the in-process node_uuids input.

Callers (DiffMerger, MergeFailureRecoverer, SchemaUpdateCoordinator,
branch tasks, CLI) receive GraphRollbacker by constructor injection; the rollback
semantics (RollbackScope, restore_metadata) are unchanged for each.

No graph-schema or index changes, no new dependencies, no API changes. Which writes
a rollback reverses (scope, window, target branch) is untouched.

Impact & rollout

  • Backward compatibility: no breaking changes; no data migration. The rollback
    is idempotent and resumable, including over state left by an interrupted run of
    the previous implementation.
  • Performance: database transaction memory is now bounded by the batch size
    instead of rollback size; application memory is O(1) (the queries return
    nothing). Wall time is comparable to the previous query and proportional to
    graph size, not merge size.
  • Config/env changes: none.
  • Deployment notes: safe to deploy.

Checklist

  • Tests added/updated
  • Changelog entry added (+rollback-metadata-restore.fixed.md)
  • External docs updated (if user-facing or ops-facing change) — n/a, internal recovery behavior
  • Internal .md docs updated (dev/knowledge/backend/merge-failure-recovery.md added and linked from backend/AGENTS.md, branch-status.md, merge-recompute.md)
  • I have reviewed AI generated content

Summary by cubic

Replaced the monolithic rollback with a two-phase, batched rollback orchestrated by GraphRollbacker to prevent OOMs and make rollbacks idempotent, resumable, and crash-safe per batch. Metadata restore now runs in-batch and correctly clears merge-time stamps for objects new to the default branch.

  • Refactors
    • Introduced GraphRollbacker with two passes: RollbackReopenEdgesQuery (reopen closed edges + optional endpoint metadata restore) and RollbackDeleteEdgesQuery (optional metadata restore, delete new edges, and remove newly orphaned vertices).
    • Removed RollbackQuery; cleanup is bundled with each phase to keep batches atomic and resumable.
    • Enforced metadata restore only on default/global branches and cleared merge-time stamps for objects first landing on default during a failed merge.
    • Updated callers to inject GraphRollbacker (DiffMerger, SchemaUpdateCoordinator, MergeFailureRecoverer, branch tasks, CLI recover, API schema, dependency builder); adjusted tests.
    • Used label-less edge scans with CALL ... IN TRANSACTIONS to reduce memory and avoid planner eager buffers; added merge-failure recovery docs and a changelog entry.