perf(diff): batch node field summary retrieval over paged queries

Merged
AJ
4mo ago
+327-59

Encountered an issue with the query to retrieve diff summaries of nodes when running merge performance testing locally on a very large branch. Could be that it was just too much data for my machine to handle, but better safe than sorry and this batching will cost us very little.

Why

Retrieving the node field summaries for a diff currently runs one query that aggregates the changed field names and node UUIDs of every changed node per kind in a single transaction. On large diffs (tens of thousands of changed nodes) that aggregation does not scale and can exhaust the Neo4j heap (Java heap space errors during merge/rebase validation).

This PR pages the retrieval over the changed nodes so memory use per query is bounded by database.query_size_limit, regardless of diff size.

Non-goals: no change to what the summaries contain or how consumers (schema-constraint validation) use them.

What changed

Behavioral changes:

  • DiffRepository.get_node_field_summaries() now issues one query per page of changed nodes instead of a single aggregate query. Results are identical; only the retrieval strategy changed.

Implementation notes:

  • EnrichedDiffNodeFieldSummaryQuery now selects one page of changed nodes (SKIP/LIMIT over a strict ORDER BY n.uuid, elementId(n)) and collects each node's altered attribute/relationship names per row, rather than aggregating across the whole diff per kind.
  • get_node_field_rows() yields one single-node NodeDiffFieldSummary per row; the repository folds them into per-kind summaries via the new NodeDiffFieldSummary.merge().
  • The DIFF_HAS_NODE match stays OPTIONAL so a diff root with no changed nodes still produces a (node-less) row. Those rows are skipped when yielding summaries; this cannot terminate pagination early because nulls sort after every real node in the ORDER BY.
  • Pagination stops when a page yields fewer nodes than the limit.

What stayed the same: the shape of NodeDiffFieldSummary (per-field node-UUID maps and derived views) and the repository's public API are unchanged.

How to review

The part deserving the most scrutiny is the pagination-termination logic: the loop breaks when a page yields fewer rows than query_size_limit, and the node-less rows from the OPTIONAL MATCH occupy page slots without being counted. This is safe only because nulls sort last in the query's ORDER BY — see the get_node_field_rows() docstring.

Impact & rollout

  • Backward compatibility: none — no API, schema, or data changes.
  • Performance: bounds per-query memory on large diffs; adds one extra round trip per query_size_limit changed nodes.
  • Deployment notes: safe to deploy.

Checklist

  • Tests added/updated
  • Changelog entry added (uv run towncrier create ...)
  • External docs updated (if user-facing or ops-facing change)
  • Internal .md docs updated (internal knowledge and AI code tools knowledge)
  • I have reviewed AI generated content