fix: remove generated template and profile schemas with their node (closes #10049)

Merged
LA
4mo ago
+296-31

Why

Loading a schema change onto a branch that removes a node with generate_template: true (via state: absent) leaves the branch permanently unable to converge: workers reloading the branch schema keep the orphaned Template<Kind> entry, crash with SchemaNotFoundError during process(), and never recover — the UI shows "Loading schemas…" forever, GraphQL on non-initiating workers misses fields from that and later loads, and every subsequent load on the branch re-triggers the failure. Reproduced on 1.10.1, 1.10.5, and current stable.

Goal: removing a node with generated kinds converges on every worker, and the branch stays usable.

Non-goals: the wider no-snapshot/no-rollback design of SchemaManager.load_schema (a mid-process() failure still leaves the in-memory registry half-updated) is a separate robustness issue and is not addressed here.

Closes #10049

What changed

Behavioral changes:

  • Removing a node or generic from a branch schema now removes the Profile<Kind> and Template<Kind> generated from it at the same time, so a worker refreshing the branch rebuilds a consistent schema instead of failing on every refresh.
  • A schema that somehow already holds a generated template with no backing node reprocesses cleanly and drops the orphan, rather than raising.
  • An object template generated for a generic is now removed with that generic. It never was before, so a deleted or renamed generic left its Template<Kind> in the generics map permanently — and in the branch hash.

Implementation notes:

The production change is 20 lines in backend/infrahub/core/schema/schema_branch.py, in two layers:

  1. SchemaBranch.delete() cascades to the generated kinds (_delete_generated_kinds). This enforces the invariant at the source: a kind leaving the schema map takes its generated kinds with it, so the orphaned state never exists. The template lookup checks self.templates and falls back to self.generics, because generate_object_template_from_node returns a GenericSchema for a generic and set() routes it into the generics map.
  2. _generate_weight_templates skips a template whose backing node is missing. generate_weight() runs before manage_object_template_schemas() in process_pre_validation, so this keeps process() total for a registry that already holds an orphan. It mirrors the guard already present in _generate_generics_templates_weight.

An earlier revision of this branch shipped layer 2 alone, after layer 1 was reverted for breaking test_exhaustive_handles_same_uuid_vertices_after_kind_migration in CI. That failure was a test-harness artifact, not a production signal, so layer 1 is back.

Keeping both layers is deliberate. Layer 1 prevents the orphan; layer 2 means a reader that meets one anyway does not crash. Making each reader tolerant instead does not scale — _generate_generics_templates_weight already needed the same guard.

What stayed the same: no schema-format or API changes, process_pre_validation() ordering untouched, profiles and templates remain excluded from the branch hash.

Suggested review order

  1. backend/infrahub/core/schema/schema_branch.py — the whole production change.
  2. backend/tests/unit/core/schema/test_schema_branch.py — the cheapest expression of both invariants.
  3. backend/tests/component/core/schema/schema_branch/test_process_idempotency.py — the save/reload behaviour.
  4. backend/tests/component/graph_traversal/test_path_traversal_query.py — two lines, per above.

How to review

The tests are the part worth scrutinising. Each layer of the fix has exactly one dedicated guard, verified by removing each layer and re-running:

Code stateWhat fails
both layers presenteverything passes
cascade removedthe two TestDeleteRemovesGeneratedKinds cascade tests (2 failed, 9 passed); the round-trip tests still pass, since process() prunes the orphan
skip removedTestProcessWithOrphanedTemplate (1 failed, 10 passed)
both removedTestSchemaBranchDbRoundtrip::test_removing_node_drops_its_generated_kinds — SchemaNotFoundError: Unable to find the schema 'TestGadget' at manager.py:1123

TestSchemaBranchDbRoundtrip saves the base schema to the database once for the class; each test then applies only its own change, on its own branch, writing a single kind. Two disjoint standalone nodes (TestGadget removed, TestDoodad flipped) keep the tests independent of execution order.

A functional test added earlier on this branch (backend/tests/functional/schemas/test_load_remove_template_node.py) was removed. Functional tests use a single shared registry (and SchemaManager), so this would not be capable for reproducing the bug. Several component tests that compare SchemaBranchs before and after updates have been added instead.

Unable to reproduce the generate_template flip causing a schema update crash

The issue states that flipping generate_template: true → false without deleting the node triggers the same failure, and concludes the trigger is "removal of the generated template, not node deletion per se". I was unable to reproduce this either in tests or against a local Infrahub app. Three independent checks:

  1. _generate_weight_templates resolves template.name, which is the node kind. A flag flip leaves the node in place, so the lookup succeeds and manage_object_template_schemas drops the template normally.
  2. On a live stack the flip converged cleanly on a clean instance while state: absent failed on that same instance. An earlier flip run on a contaminated instance did time out, but zero error lines named the flip branch and all 24 named the earlier state: absent branch — the sync check is global, so one wedged branch keeps it red. The issue's own steps run the removal first, which is the likely origin of the observation.
  3. At infrahub-v1.10.1, _generate_weight_templates, delete(), and the removal handling in manager.py are materially identical to stable, so the flip could not have failed that way there either.

test_disabling_generate_template_drops_template covers the flip anyway, since the issue lists it under Expected Behavior — but it passes with both layers removed, so it is a convergence guard, not a regression test for this bug.

Testing

Verified end to end on a local deployment with 1 API worker and 2 task workers, same schemas and steps against both versions:

Scenariostable @ c918af76ethis branch
Remove a generate_template node (state: absent) on a branch"Schema is still not in sync"; both workers looping SchemaNotFoundError: Unable to find the schema 'DemoDoohickey'Schema updated on all workers
Subsequent additive load on that same branchn/a — branch already wedgedSchema updated on all workers
Flip generate_template: true → false on a branchconverges (never the bug)Schema updated on all workers
SchemaNotFoundError in task-worker logs18 branch-tagged failures0

Impact & rollout

  • Backward compatibility: none broken. No API or schema-format changes. A worker whose registry is already poisoned self-heals on its next schema refresh.
  • Performance: negligible — two dict lookups per delete, and a try/except per template during weight generation. The component test consolidation cut that file's database work from three full save + reset + core-model-registration cycles to one (70.8s for the file).
  • Config/env changes: none.
  • Deployment notes: safe to deploy. Clean backport candidate for the 1.10.x line — the touched code is unchanged since 1.10.1.

Checklist

  • Tests added/updated
  • Changelog entry added (changelog/10049.fixed.md)
  • External docs updated (if user-facing or ops-facing change) — N/A
  • Internal .md docs updated (internal knowledge and AI code tools knowledge) — N/A
  • I have reviewed AI generated content — pending human review

🤖 Generated with Claude Code