fix(backend): delete branch data in bounded batches
Why
Deleting a large branch can fail with Neo.TransientError.General.MemoryPoolOutOfMemoryError. Because the delete commits its writes in batches while the surrounding query does not, the failure leaves the branch stranded in DELETING — filtered out of the branch list, so invisible to the user — with most of its data still in the graph, taking up space with no way to reclaim it.
Goal: make a branch delete complete regardless of how large the branch is, by bounding how much transaction memory it needs — and clean up the branches previous failures already stranded.
Non-goals: no change to what a branch delete removes.
Closes #9889
What changed
Behavioral changes:
- A branch delete no longer exhausts transaction memory on large branches. During a local testing, deleting a branch of 1,074,217 edges completed at a 512 MiB transaction limit, where the previous implementation ran out of memory.
- It is also faster: 27.8s vs 87.9s on that branch at the default limit.
- A new graph migration (
075_finish_deleting_branches) finishes the job for branches an earlier failure abandoned. Upgrading finds every branch left inDELETING, runs it through the new deleter, and removes the branch node — reclaiming space that was otherwise unreachable, since such a branch cannot be looked up or deleted again through any normal path. Branch.delete()now raisesNotImplementedErrorpointing atBranchDeleter. It is kept as an override rather than removed, because the inheritedStandardNode.deletewould drop theBranchvertex and silently orphan every edge and vertex on the branch — a worse failure than the one being fixed.- A delete that failed part way through can now be retried. Both the
BranchDeletemutation and thebranch-deleteflow load the branch withignore_deleting=False, so theDELETINGstatus the first attempt leaves behind no longer hides it from the lookup. Previously the retry failed withBranchNotFoundErrorand the migration was the only way out — and it runs once, on upgrade, so any later failure stranded the branch for good. Two consequences to be aware of: the mutation will now accept a delete for a branch whose delete is still running (every query involved is idempotent, but nothing serialises the two runs), and a retry re-emitsBranchDeletedEvent. Making the surrounding flow steps idempotent is not addressed here.
Implementation notes:
The single mega-query became a set of bounded queries sequenced from Python by a new BranchDeleter component:
DeleteBranchAgnosticRelationshipsQuery/DeleteBranchAgnosticAttributesQuery— the agnostic peers of nodes that exist on no other branch.DeleteBranchEdgesQuery— one batch of edges of a single relationship type, re-run until no edges of that type remain, for each member ofDatabaseEdgeType.
Three things drove the design:
- Peak transaction memory is now a function of batch size, not branch size. The old query's dominant allocation was
WITH DISTINCT elementId(s), elementId(d)followed bycollect(s_id) + collect(d_id): an eagerDISTINCTover pairs (so a node was stored once per edge it had), materialized as element-id strings, then twocollects plus a concatenated copy held at once. The existingCALL {} IN TRANSACTIONSonly bounded the inner writes — everything upstream lived in one outer transaction, which is why the earlier "use smaller transactions" fix didn't help. The replacement has no eager aggregation beyond aDISTINCTbounded by the batch. - Naming the relationship type lets the
branchrange index serve the match.MATCH (s)-[r]->(d) WHERE r.branch = $branch_namecan use none of the per-type indexes incore/graph/index.py, so it scanned every edge in the database. One query per type turns that into an index seek, and because deleted entries leave the index, each batch is O(batch) rather than a re-scan from the start. - Each batch deletes the vertices its own edge deletions left bare, so no vertex bookkeeping is needed anywhere. This is sound because every branch edge is removed by the batch's
DELETEand both endpoints are re-examined afterwards — a vertex is examined once per edge it had, so the batch that removes its last edge is the one that sees it at degree zero. TheDISTINCTis load-bearing: it forces the batch's edge deletes to complete before the first vertex is examined. It must not become aDETACH DELETE, which would remove branch edges that then never reach a batch of their own, leaving the vertices on their far side unexamined and orphaned.
A fourth point applies to the agnostic cleanup specifically. Its batches count Nodes, and one Node can drag an unbounded number of peer vertices into the transaction with it, unlike the edge batches where one row is one edge. So it takes min(batch_size, 500) rather than the configured batch size: 500 is what this phase used before, and it is a ceiling rather than a fixed size so that an operator who lowers the configured limit to fit a constrained database is not handed a larger batch here than they asked for.
The migration reuses the delete path rather than reimplementing it: it loads each stranded branch with Branch.get_by_name(..., ignore_deleting=False) — the escape hatch that makes these branches reachable at all — and hands it to BranchDeleter.delete(), so it exercises exactly the code a normal delete does. Finding them needs its own query because the shared branch list query filters DELETING out unconditionally, so asking it for those branches returns nothing.
What stayed the same:
- No node-schema changes, no GraphQL or API changes. The only migration is the graph migration above; it adds and alters nothing, it only finishes deletes.
- The set of vertices removed is unchanged — 416,021 on the benchmark branch, identical to before, with pre-existing orphans left untouched.
- Phase 1 (agnostic cleanup) keeps its existing Cypher and must still run first: both queries find their candidate nodes through the branch's
IS_PART_OFedges, which phase 2 destroys. That ordering used to be implicit in one query text and is now spread across Python control flow, so it is stated in the docstrings — and it is what makes an interrupted delete safe to re-run. Migration032still works, viaBranchDeleter.delete_branch_data(), which takes a branch name rather than aBranchbecause the branches it cleans up have noBranchvertex left.
Alternatives considered:
- Accumulating touched vertex ids client-side and sweeping at the end. Rejected: millions of ids held in Python.
- Marking candidates in the database with a label. Rejected: it writes to the store and the token index where there was only a read, and a crashed run leaves labels behind to poison the next delete.
- A final label-scoped orphan sweep. Implemented and measured, then removed once
DETACH DELETEwas dropped: it cost a full 9M-vertex scan per delete and also deleted pre-existing orphans, a scope change the current version doesn't make. - Wrapping the per-type query in
CALL {} IN TRANSACTIONSinstead of a PythonLIMITloop. Not taken: it would stream a scan over the same index its committed batches are mutating, while deleting vertices mid-scan. TheLIMITloop gives each batch a fresh snapshot for the price of one round trip per batch.
One deliberate gap: IS_RESERVED is the only member of DatabaseEdgeType without a branch range index, so its pass is a type scan rather than a seek. In practice it is only ever written on the global branch, so a branch-scoped delete never matches it. Adding the index is a DB index change and was left out of this PR.
Local performance testing results
Results — branch of 1,074,217 edges in a 24M-edge database:
| before | after | |
|---|---|---|
| 4.11 GiB memory limit | 87.9s | 27.8s |
| 512 MiB memory limit | OOM at 28.8s, branch stranded | 31.3s, OK |
| vertices removed | 416,021 | 416,021 |
| new orphaned vertices | 0 | 0 |
edges on main | unchanged | unchanged |
Impact & rollout
- Backward compatibility:
Branch.delete()now raises. The only production caller was thebranch-deleteflow, and every caller in this repo is updated; any out-of-tree caller (Infrahub Enterprise, scripts) must switch toBranchDeleter. - Performance: 3.2× faster on the benchmark branch, and peak transaction memory no longer scales with branch size. Cost is one extra round trip per batch, negligible against the work each batch does.
- Config/env changes: none. Batch size comes from the existing
database.query_size_limitsetting, which the agnostic-cleanup phase caps at 500 for the reason above. - Deployment notes:
GRAPH_VERSIONgoes to 75, so this requires the usualinfrahub db migrateon upgrade. The migration is a no-op on any database with no branch inDELETING; where there is one, it deletes that branch's data, so expect it to run for as long as a branch delete would — it reports each branch as it starts and the edge count when it finishes, sinceinfrahub db migratesuppresses the deleter's own logging and silence would otherwise look like a hang. A branch that cannot be deleted is named in the migration's errors and does not stop the others. Safe to re-run: an interrupted delete simply resumes.
Checklist
- Tests added/updated
- Changelog entry added (
uv run towncrier create ...) - External docs updated (if user-facing or ops-facing change) — n/a, no user-facing change beyond the fix itself
- Internal .md docs updated (internal knowledge and AI code tools knowledge)
- I have reviewed AI generated content