feat(computed-attribute): selective recompute of transform-based computed attributes on git updates

Merged
GM
4mo ago
+2995-97

Why

On any commit to any linked git repository, every Python transform-based computed attribute is recomputed for every node of its kind, even when the commit touched nothing that feeds that attribute. The commit event carries no diff, so the recompute falls back to a full sweep. On large datasets this is slow and wasteful, and it buries the work that actually matters.

Goal: recompute a Python transform-based computed attribute only when the transform that feeds it actually changed.

Non-goals: artifact-definition and generator consumers of the fingerprint (separate tickets), dropping .infrahub.yml from the dependency closure (IFC-2775), and Jinja2 computed attributes (git-independent, already skip via template_hash).

Closes IFC-2804

What changed

Behavioral:

  • A commit now recomputes only the computed attributes whose Python transform actually changed, not every transform-based attribute on the branch.
  • The change signal is the branch-aware fingerprint attribute from IFC-2844. An unchanged fingerprint means no update event and no recompute. An edit and its exact revert produce the same fingerprint, so they trigger no work.
  • A transform that declares no watch keeps per-commit recompute, but scoped to its own attributes only (the safe over-regenerate default).

Implementation:

  • Recompute is driven by the transform node's own lifecycle instead of the commit event. Three triggers on CoreTransformPython (created, updated-on-fingerprint, deleted) replace the old commit trigger.
  • The fired flow resolves the transform to the attributes it feeds (by name or id) and recomputes only those, reusing the existing recompute fan-out.
  • The lifecycle flow also reconciles the node-input recompute automations on every event. That reconciliation used to happen as a side effect of the removed commit trigger, so it runs here in a finally and never gets skipped when the recompute leg fails. This is what keeps a transform-only import from silently leaving those automations unbuilt.
  • Loop safety comes from the kind and field match (recompute writes land on the attribute's own node kind, never on the transform) and from the live origin match (merge and rebase replays are excluded).

What stayed the same:

  • No schema change (the fingerprint attribute already exists from IFC-2844), no migration, no generated files.
  • The schema-change recompute path and the merge/rebase recompute path are untouched.
before:  commit event (no diff)          -> recompute EVERY transform attr x EVERY node
after:   CoreTransformPython fingerprint  -> recompute only that transform's attrs

Suggested review order

  1. dev/specs/ifc-2804-selective-recompute/ (spec, research, plan, contracts) for the intent and the decisions.
  2. backend/infrahub/computed_attribute/recompute_resolution.py (the pure resolver).
  3. backend/infrahub/computed_attribute/tasks.py (process_transform_lifecycle and the shared reconcile helper).
  4. backend/infrahub/computed_attribute/triggers.py plus the two catalogue swaps.
  5. Tests: unit, then component, then integration.

How to review

The load-bearing correctness piece is the reconcile-on-every-event in the lifecycle flow. It guards against under-regeneration on transform-only imports (a code review of the MVP caught the hole where removing the commit trigger would otherwise strand the node-input automations). Extra scrutiny there is welcome.

dev/specs/ifc-2804-selective-recompute/review-followups.md lists the deferred low-severity items (a reconcile lock, an idempotent double-recompute overlap on first import, and a rename-by-name edge case).

How to test

# unit
uv run pytest backend/tests/unit/computed_attribute/ -q
# component (testcontainers)
uv run pytest backend/tests/component/computed_attribute/test_transform_lifecycle_recompute.py --neo4j -q
# integration (testcontainers)
uv run pytest backend/tests/integration/git/test_selective_recompute.py --neo4j -q

All of the above pass locally, along with uv run invoke lint (ruff, ty, mypy) and uv run invoke docs.validate.

Impact & rollout

  • Backward compatibility: fingerprints are null until each repository's first import after upgrade. That first import stamps a fingerprint and recomputes each transform's attributes once. This is a one-time cost per transform, not a per-commit sweep.
  • Performance: recompute work now scales with the number of changed transforms per commit, not the total number of computed attributes on the branch.
  • Config/env changes: none.
  • Deployment notes: safe to deploy.

Checklist

  • Tests added/updated
  • Changelog entry added
  • External docs updated (if user-facing or ops-facing change) - not needed, internal recompute behavior, covered by the changelog
  • Internal .md docs updated (dev/specs and review-followups)
  • I have reviewed AI generated content

Summary by cubic

Recomputes Python transform-based computed attributes only when their transform actually changes, replacing the commit-wide sweep with transform lifecycle triggers. First import after upgrade stamps each transform’s branch-aware fingerprint and recomputes its attributes once.

  • New Features

    • Three CoreTransformPython lifecycle triggers (create/update-on-fingerprint/delete) run computed-attribute-process-transform-lifecycle; the old commit trigger is removed.
    • Branch-aware fingerprint: unchanged or edit+revert does nothing; transforms without watch keep per-commit recompute scoped to themselves.
    • Live-only: filters on origin=live to ignore merge/rebase replays; waits for schema convergence; tolerates a missing transform.
  • Bug Fixes

    • Resolves transforms by name and UUID and dedupes by (kind, attribute) to avoid double or missed recomputes.
    • Always reconciles node-input automations in finally, gathering once under a single registry lock to prevent race-induced drops.