queries for node-scoped uniqueness constraints

Merged
AJ
4mo ago
+1325-3

Why

When uniqueness validation fires from a data change (proposed-change checks, merges, rebases), the
UniquenessChecker scans the entire population of each implicated kind via
NodeUniqueAttributeConstraintQuery — cost tracks how many objects exist, not how many changed.
The scan also returns every population-wide duplicate (>100k rows on that dataset) for Python-side
post-processing, including violations that pre-date the change.

Goal: provide the query and component foundation to validate only the changed nodes: a
batched, index-anchored query whose cost tracks the size of the change, plus the pieces that
identify which nodes a data diff implicates.

Non-goals: this PR does not yet wire any of it. UniquenessChecker still runs the full scan;
the determiner does not yet emit changed-node uuids. The checker integration, the diff-side scoping,
and generic/implementation dedup are stacked follow-ups. Nothing in this PR changes runtime
behavior.

Part of IFC-2796. Later PR(s) will actually wire the changes into the tasks and test them.

What changed

Behavioral changes: none — every addition is unused by production code paths in this PR.

New pieces (grouped by role in the upcoming targeted path):

  • TargetedUniquenessValidationQuery (validators/uniqueness/query/targeted_validation.py) —
    one Cypher query per (kind, constraint group) validating ALL changed nodes against the full
    population in a single round trip. It resolves the first constraint element's value for the
    changed nodes only, probes the population for other nodes sharing it through the
    AttributeValueIndexed/Node(uuid) indexes, drops changed nodes with no match, and resolves each
    subsequent element only for the survivors — a changed node whose first element collides with
    nothing never has its remaining values read.
  • AffectedUniquenessDependentsQuery + UniquenessDependentResolver
    (query/affected_dependents.py, dependent_resolver.py) — cross-kind resolution: when a
    constraint reads a peer's attribute (e.g. owner__name), a change to the peer kind implicates
    nodes of the constrained kind; this resolves the changed peers into those nodes. Each hop resolves
    to a single winner across the input/default/global branches (latest edge on the deepest branch),
    so a relationship deleted on the input branch excludes the node — matching what the merge will
    produce — while the default branch's post-fork state decides wherever the input branch is silent.
  • NodeDiffIndex (validators/node_diff_index.py) — per-kind index over a data diff's changed
    attributes, relationships, and node uuids; NodeDiffFieldSummary gains the node_uuids field it
    reads.
  • Query package restructure: validators/uniqueness/query.py moves to
    validators/uniqueness/query/validation.py inside a new package whose __init__ re-exports
    everything — existing import paths are unchanged.

Design decisions worth knowing:

  • Peer-attribute constraint elements (owner__name) are rejected by the targeted query
    (ValueError): schema updates disallow attributes-of-relationships in uniqueness_constraints
    today, so the targeted path never legitimately receives one.

What stayed the same: NodeUniqueAttributeConstraintQuery and UniquenessValidationQuery are
untouched (the latter still serves the node-save constraint in
core/node/constraints/grouped_uniqueness.py); UniquenessChecker behavior is unchanged; no DB
schema or migration changes.

Performance

Regarding the new targeted uniqueness query, on a large local dataset (≈500k DcimInterface nodes),
a single uniqueness-constraints scan ran for 8+ minutes and then OOM-killed Neo4j. A sibling kind's
scan ran for 30 minutes before being killed.

Benchmarked against a large generated dataset (2%/5%-of-population change branches): the targeted
query validates 9,400 changed nodes of a 503k-node kind in ~12s and 23,500 in ~29s, where the
current full scan of the same kind ran 8.4 minutes before OOM-killing Neo4j (and >30 minutes on a
464k-node sibling kind before being killed). All 31 implicated kinds complete in ~32s / ~60s total.

Impact & rollout

  • Backward compatibility: no behavior change — additions only, plus an import-transparent file
    move. API contract unchanged.
  • Performance: no actual changes in this PR.
  • Config/env changes: none.
  • Deployment notes: safe to deploy.

Checklist

  • Tests added/updated
  • Changelog entry (deferred to the follow-up PR that wires the behavior change)
  • External docs updated (n/a — no user-facing change yet)
  • Internal .md docs updated (n/a)
  • I have reviewed AI generated content

Summary by cubic

Adds index-backed, node-scoped uniqueness validation that checks only changed nodes and their dependents instead of scanning all nodes. Supports IFC-2796 and reduces validation time on large kinds; honors branch visibility, post-fork default updates, and migrated-kind duplicates.

  • New Features

    • TargetedUniquenessValidationQuery validates only given node UUIDs for one constraint, comparing attributes and 1:1 relationships while enforcing node liveness and branch visibility; supports large attribute types (no index), generic kinds across implementations, and ignores stale same-UUID duplicates from kind migrations.
    • Added UniquenessDependentResolver (uses AffectedUniquenessDependentsQuery) to resolve constrained nodes referencing changed peers; added NodeDiffIndex and extended NodeDiffFieldSummary with node_uuids to drive scoping.
  • Refactors

    • Moved uniqueness queries under core/validators/uniqueness/query and re-exported to keep imports stable.
    • Fixed a value check so falsy attribute values compare correctly.