add inherited attrs during schema migration

Merged
AJ
3mo ago
+839-27

Why

When an existing node kind starts inheriting from a generic, the newly-inherited attributes are never created on pre-existing nodes in the database. Reads will return id: null, is_default: true, update mutations will report ok: true while persisting nothing, and attribute filters will never match. Nothing in the schema-diff pipeline owns the "a node just gained inheritance" event — a brand-new generic lands in diff.added (which emits no migrations), an existing generic gaining an inheritor never appears in diff.changed (used_by is excluded from the diff), and the node-scoped node.attribute.add migration deliberately skips inherited attributes (the #7407 duplicate-row protection).

This PR makes the kind-update migration — the one that already owns relabeling the node vertices — also create the attribute rows, and makes the schema-migration batch race-free by ordering kind-update migrations before everything else.

Non-goals:

  • Healing installs already damaged before this fix — that is a follow-up PR adding an upgrade-time graph migration, planned for the same release (dev/specs/ifc-2619-inherited-attr-migration/).
  • Refactoring the migration-within-a-migration pattern shared with node_uniqueness_constraints_update.py (follow-up issue).
  • Stale Template{generic} labels on pre-existing template vertices — a distinct pre-existing gap (follow-up issue).

Part of #9284 (first of two PRs; the issue closes with the healing migration). Jira: IFC-2619.

What changed

Behavioral changes:

  • Loading a schema in which an existing kind newly inherits a generic now creates real attribute rows on all pre-existing nodes of that kind, with the generic's default values, including Profile{kind} and Template{kind} instances where the attribute supports them (unique/read-only attributes land on nodes only, matching how profile/template schemas are generated).
  • Inherited NumberPool attributes are allocated: each pre-existing node gets a distinct number, drawn from a single pool registered against the generic's kind — no duplicate pools, no duplicate allocations.
  • Schema migrations now run in two phases: all kind-update migrations (node.inherit_from.update, node.name.update, node.namespace.update) complete before any other migration starts because the kind update migrations involve duplicating Node vertices. If other migrations ran concurrently with a kind update, it is possible that the kind update would basically overwrite the changes of the other migration(s). The second phase is skipped when the first reports errors.

Implementation notes:

  • NodeKindUpdateMigration.execute() runs a NodeAttributeAddMigration per newly-inherited attribute after the vertex duplication commits, reusing the existing (acknowledged-ugly) migration-within-a-migration pattern from node_uniqueness_constraints_update.py. Reusing the whole migration is what makes Profile/Template targeting and NumberPool allocation come along for free.
  • NodeAttributeAddMigration gains a force_inherited opt-in. The default guard is unchanged, so the #7407 case (a generic gains an attribute → the single generic-scoped run covers all inheritors) keeps its duplicate-row protection.
  • Kind-update migrations execute with max_concurrent_execution=1: one schema load can emit several kind-update migrations for the same kind (e.g. a rename plus an inherit_from change), and two concurrent duplications over overlapping vertices each create replacement vertices. The idempotency guards only protect sequential reruns.
  • Atomicity across duplication + attribute creation is deliberately not transactional (nested transactions are not supported); both queries are idempotent and schema_path_migrate has retries=3, so a partial failure heals on rerun — covered by a dedicated test.
  • split_migrations_by_phase derives the phase-1 name set from MIGRATION_MAP rather than hard-coding it, and is a pure function with its own unit tests.

Case coverage:

Schema changeCovered by
Generic gains an attribute, node already inheritsphase 2, generic-scoped (unchanged)
New generic + existing node inherits itphase 1, kind-update sub-migrations
Existing generic + node newly inheritsphase 1, kind-update sub-migrations
Both at once (compound)phase 1 for that node; phase 2 no-ops on it via idempotency, covers other inheritors

What stayed the same: no schema-model or API changes, no new graph migration or GRAPH_VERSION bump, no change to how the schema diff is computed (backend/infrahub/core/models.py untouched), and the generic-gains-attribute path is byte-for-byte the same.

Suggested review order

  1. backend/infrahub/core/migrations/schema/node_attribute_add.py — the force_inherited flag and the guard it relaxes.
  2. backend/infrahub/core/migrations/schema/node_kind_update.py — the execute() override and _newly_inherited_attributes().
  3. backend/infrahub/core/migrations/schema/tasks.py — two-phase batching and the serialized phase 1.
  4. Tests: unit (tests/unit/core/migrations/schema/test_tasks.py), component (tests/component/core/migrations/schema/test_node_kind_update.py, test_node_attribute_add.py), integration (tests/integration/schema_lifecycle/test_schema_add_inherited_generic.py).

How to review

Focus on the guard semantics in node_attribute_add.py: the invariant is that the generic-scoped run remains the sole owner of "generic gains an attribute" while the kind-update path becomes the sole owner of "node gains a generic" — the compound case relies on AttributeAddQuery's idempotency where the two overlap, exercised end-to-end by step 04 of the integration test (existing generic onto an existing kind while both the kind and the generic gain a new attribute in the same load).

The phase split in tasks.py is small but load-bearing: the integration test's compound step is only deterministic because kind-updates complete first and run serialized.

Impact & rollout

  • Backward compatibility: no breaking changes. Behavior changes only at schema-load time for loads that add inheritance to an existing kind — previously silently broken, now correct. Installs already damaged before upgrading are repaired by the follow-up healing-migration PR targeted at the same release.
  • Performance: kind-update migrations now run one at a time; there is at most one per changed kind per load, so the serialization is negligible. Phase 2 keeps the previous concurrency.
  • Config/env changes: none.
  • Deployment notes: safe to deploy; no database migration, no SDK coordination.

Checklist

  • Tests added/updated
  • Changelog entry added (uv run towncrier create ...)
  • External docs updated (if user-facing or ops-facing change) — bug fix restoring documented behavior; no doc changes needed
  • Internal .md docs updated (internal knowledge and AI code tools knowledge)
  • I have reviewed AI generated content

Summary by cubic

Creates real attribute rows for newly inherited generic attributes during schema migrations so updates persist and filters work. Addresses IFC-2619 by materializing defaults (incl. NumberPool allocations) and running kind-update migrations first and sequentially to prevent races.

  • Bug Fixes
    • Run NodeKindUpdateMigration first and sequentially; then run other migrations (two-phase batching).
    • During kind updates, add each newly inherited attribute via NodeAttributeAddMigration with force_inherited=True.
    • Add force_inherited to NodeAttributeAddMigration; default still skips inherited attrs to avoid duplicates.
    • Materialize attributes for existing nodes, profiles, and templates; allocate NumberPool from the generic’s pool.