feat(regeneration): safe-fallback escalation and post-merge generator cascade for selective regeneration

Merged
PO
4mo ago
+2043-323

Summary

Selective post-merge regeneration (IFC-2704) for artifact and generator definitions: the merge
follow-up regenerates only what a merge changed when the change signal is trustworthy, and falls
back to full regeneration whenever the signal cannot be trusted. This PR adds the safe-fallback
escalation, the generator → artifact cascade, and the isolated artifact member/limit filter, on top
of consolidated test fixtures.

Supersedes #9997 (same branch, history rewritten into logical commits; no content change).

Changes

AreaBehavior
Untrusted-signal fallbackA definition with no computed fingerprint escalates its whole repository to full regeneration; a null or incomplete dependency closure regenerates that definition in full. The escalation is computed across both definition kinds, so a repository is escalated as a whole, and self-heals once its fingerprints are populated on the next import.
Reason-code modelThe free-text predicate reason becomes a RegenerationTrigger pairing a machine-comparable RegenerationReason with its log line; the dependency-closure classification is a reusable predicate. A fingerprint change on a definition node reads as a code-input change.
Generator → artifact cascadeA generator dispatched by the follow-up runs after the merge diff was captured, so its output is absent from that diff. Every selected generator is awaited, then all artifacts are regenerated, so no consuming artifact is left stale. This holds identically for direct and proposed-change merges.
Artifact member/limit filterThe member/limit selection is isolated into RequestArtifactDefinitionGenerate.selects_member so the limit boundary is covered directly.
Test fixturesShared SDK diff-summary factories across the suites; the WorkflowRecorder holds one ordered call log with per-kind views derived from it.
DocsChangelog fragment and a performance-scenarios document recording which merges take the selective path and the conditions that gate it.

Commit layout

Seven logical commits, each independently reviewable:

  1. test(regeneration): shared diff-summary fixtures and single-source WorkflowRecorder
  2. refactor(regeneration): carry a machine-comparable reason code on predicate outcomes
  3. feat(regeneration): over-regenerate when a definition's change signal is untrusted
  4. feat(regeneration): cascade artifact regeneration after a post-merge generator
  5. refactor(git): extract the artifact member and limit filter
  6. test(regeneration): cover selective post-merge regeneration behavior
  7. docs(regeneration): changelog fragment and spec artifacts

Working scenarios

The backend unit suite passes (1922 tests). Selection is covered at the unit and component tiers
(real-graph selection driving the dispatcher through a recording workflow backend). The
testcontainers integration scenarios below were validated locally against rendered artifact content;
those slow tests land separately.

ScenarioResult
Generator → artifact cascade (both merge kinds)✅ artifacts reflect the generator's output after the merge
Generator opt-out (execute_after_merge disabled)✅ the generator is left untouched
Updated transform on main✅ renders against main's current transform, not the branch's stale copy
New definition on main✅ picked up and regenerated by the branch merge
Diverged main (main gained a member)✅ the branch member regenerates against the merged state
Member narrowing✅ only the changed member regenerates; an irrelevant-kind change regenerates nothing; deleting a member does not break the path
Content composition refresh❌ known limitation, out of scope: a composite artifact that inlines an upstream artifact is not refreshed after a merge that changes the upstream

How to review

  1. Reason-code model — backend/infrahub/core/regeneration/{models,predicates}.py.
  2. Untrusted-signal escalation — backend/infrahub/core/merge/selective_regen/fallbacks.py, definition_selector/, orchestrator.py.
  3. Generator cascade — backend/infrahub/core/merge/regeneration_dispatcher.py.
  4. Artifact member/limit filter — backend/infrahub/git/models.py, git/tasks.py.
  5. Tests — unit (tests/unit/core/regeneration/, tests/unit/core/merge/, tests/unit/git/) → component/integration (tests/*/proposed_change/test_merge_selective_regen.py).
    6 Spec & docs — dev/specs/ifc-2704-incremental-merge-regen/ + changelog fragment.

Notes

  • Draft: the testcontainers integration tests are held out for now.
  • The follow-up behaves identically on direct and proposed-change merges, so no merge-kind flag is threaded through the dispatch chain; the cascade decision depends only on whether any generator was selected.

Deferred to #10026 — artifact regeneration after a generator cascade

Whenever an after-merge generator is selected, the follow-up currently regenerates every artifact
on the branch (the blanket TRIGGER_ARTIFACT_DEFINITION_GENERATE). This is deliberate over-execution:
the generator's output lands after the merge diff was captured, so the consuming artifacts cannot be
identified from that diff. Narrowing this to only the artifacts that consume the generator's output —
and sequencing it correctly in the error/fallback path — is tracked in #10026:

  • Precision. Any selected generator forces a full artifact fan-out; enabling the feature no longer guarantees narrowed work when after-merge generators are in play.
  • Fallback sequencing. If an awaited generator run fails, dispatch falls back to the blanket full regeneration, which submits the generator and artifact triggers concurrently (no sequencing) — the same behavior as every other fallback path. The happy-path sequencing guarantee does not hold on that error path.

Summary by cubic

Implements selective post-merge regeneration for generator and artifact definitions (IFC-2704). Falls back safely to full regeneration when change signals are untrusted, and cascades artifact regeneration after generators to avoid stale outputs.

  • New Features

    • Selective post-merge regeneration narrows execution to changed definitions and impacted members; opt-in via env INFRAHUB_SELECTIVE_EXECUTION_AFTER_MERGE (default off).
    • Safe fallbacks escalate to full regeneration when signals are untrusted (missing fingerprint, null/incomplete dependency closure), computed repository-wide across both definition kinds.
    • Generator→artifact cascade awaits selected generators, then regenerates artifacts; respects execute_after_merge opt-out and falls back to blanket full regeneration on errors.
  • Refactors

    • Introduced RegenerationReason and RegenerationTrigger; added a reusable dependency-closure classifier and treat fingerprint changes as code-input changes for clearer logs.
    • Extracted member/limit filtering into RequestArtifactDefinitionGenerate.selects_member to correctly include members without artifacts; consolidated test fixtures and WorkflowRecorder with reset; added integration tests and performance/docs updates.