scope uniqueness constraints to affected objects

Merged
AJ
4mo ago
+1555-357

Why

Uniqueness validation scanned the entire population of a kind on every data change. Merging, rebasing, or proposing a change that touched a handful of nodes of a large kind still ran a population-wide scan — slow on big kinds (hundreds of thousands of nodes) and it shipped large duplicate result sets back to Python.

Goal: scope a data-triggered uniqueness check to only the nodes the diff actually affects, so the work is proportional to the size of the change rather than the size of the kind.

Non-goals: this does not change node-save-time uniqueness (grouped_uniqueness), does not touch other constraint validators, and does not change the uniqueness error surfaced to users.

Note: uniqueness constraints do not support attribute of relationships (e.g. "device__name"). some parts of the code support this kind of uniqueness constraint, but not all of it and it is still blocked when creating/updating a schema.

Planned follow-ups

  • refactor MergeConstraintValidator construction. schema_branch should probably be an argument for this class and its dependents, not a dependency
  • check on integration/functional-level testing for uniqueness constraint validation and add more if necessary
  • issue for supporting attributes on relationships in uniqueness constraints

Part of IFC-2796 (epic IFC-2706).

What changed

Behavioral

  • Data-triggered uniqueness validation (merge, rebase, proposed-change integrity) now validates only the changed/affected nodes via a batched, index-anchored targeted query instead of a full-population scan.
  • A uniqueness check is scoped to exactly the nodes whose changed field participates in the constraint — changing one node's unique field no longer drags in every other changed node of the kind.
  • A newly added or broadened constraint (schema-diff origin, with no affected-node set) still runs the full-population scan, as before.
  • Cross-kind constraints that read a peer's attribute (e.g. owner__name) fall back to full-population validation, which supports them; the targeted query does not. Again, this type of uniqueness constraint cannot be added to schemas currently.
  • Uniqueness error messages are unchanged — attribute values render identically to the full-population path regardless of their type.

Implementation notes

  • A node_uuids carrier is threaded end to end: None means full scan, a list means validate exactly those nodes. It rides from the diff summary through the determiner, the constraint info, and the validator request into the checker.
  • NodeDiffFieldSummary/NodeDiffIndex now keep the correlation between each changed field and the nodes that changed it (per-field UUID maps), so scoping targets a specific field's nodes rather than every changed node of the kind.
  • New here (constructor-injected, wired by build_constraint_validator_determiner): UniquenessConstraintScoper decides, per kind, whether uniqueness must run and which nodes it affects (None → full scan); the ConstraintValidatorDeterminer update threads the affected set through; constraint dedup/merge helpers collapse a node's uniqueness check into its generic's where the generic already covers it.
  • UniquenessChecker now runs the targeted query for a scoped set, and falls back to the full-population scan when a constraint reads a peer attribute.

What stayed the same

  • The full-population path and the node-save-time uniqueness checker (grouped_uniqueness) are untouched.
  • No GraphQL/API contract changes, no database schema or migration changes.

Suggested review order

  1. core/diff/model/path.py, core/diff/query/field_summary.py, core/validators/node_diff_index.py — how per-field changed-node UUIDs are captured.
  2. core/validators/uniqueness/scope.py + core/validators/determiner.py — how the affected-node set (or None) is decided.
  3. core/validators/uniqueness/checker.py — running the targeted query for a scoped set vs the full-population fallback.
  4. proposed_change/tasks.py, merge/constraints.py, branch/tasks.py — the three call sites that thread the affected set through.
  5. Tests.

How to review

Focus on the scope decision (UniquenessConstraintScoper._compute_scope) and the checker's targeted-vs-full-population branch (UniquenessChecker.check / _supports_targeted) — those govern correctness. Mechanical: the attribute_node_uuids conversions across the test files.

Impact & rollout

  • Backward compatibility: no breaking changes; uniqueness error messages are identical to before.
  • Performance: the headline win. On a large local dataset (≈500k-node kind) the full-population scan ran 8+ minutes before OOM-killing Neo4j; the targeted query validates ~9,400 changed nodes of that kind in ~12s and ~23,500 in ~29s, and all implicated kinds complete in tens of seconds. Work is now bounded by the change, not the population.
  • Config/env changes: none.
  • Deployment notes: safe to deploy; no coordinated release or migration required.

Checklist

  • Tests added/updated
  • Changelog entry added
  • External docs updated (no user-facing surface change)
  • Internal .md docs updated (internal knowledge and AI code tools knowledge)
  • I have reviewed AI generated content

Summary by cubic

Scope node-level uniqueness validation to only the nodes that changed each unique field (and their dependents), using diff-derived IDs and batched, index-backed queries. Full scans now run only when a constraint is added/broadened or when peer-attribute elements are present, cutting worst-case merge and schema-update time (IFC-2796).

  • Bug Fixes

    • Fall back to full-population validation for peer-attribute paths (e.g., owner__name) unsupported by the targeted query.
    • Reset scoper cache on determiner re-init to prevent stale scopes between diffs.
    • Stringify targeted values in error paths to match full-scan messages.
  • Refactors

    • Batch checks per constraint group via TargetedUniquenessValidationQuery, paging over changed nodes with one read-only session per change.
    • UniquenessChecker reads branch/schema from the validator request; dependency builder and m018 migration updated.
    • Carry node_uuids through SchemaUpdateConstraintInfo and validator requests across proposed-change, merge, rebase, and migrations; merge/dedup constraints and standardize on ConstraintIdentifier.NODE_UNIQUENESS_CONSTRAINTS_UPDATE.
    • Consolidate chunking into infrahub.utilities.chunks.chunked and replace local helpers.
    • Refactor UniquenessConstraintScoper for clarity; trim docstrings.