feat(regeneration): target artifacts from an after-merge generator's output

Merged
PO
4mo ago
+946-177

Summary

Refines the post-merge generator → artifact cascade: instead of regenerating every artifact once
a generator runs, regenerate only the artifacts its output affects.

Change

A generator dispatched by the follow-up runs after the merge diff was captured, so its writes are
absent from that diff. Once every generator has run, this diffs the target branch over the window
they ran in and selects only the artifacts that read the changed data (reusing the artifact half of
the merge selector). Full regeneration stays the fallback if that output can't be captured or
selected, so no artifact is left stale. Direct and proposed-change merges take the same path.

How tested

LevelCoverage
Unittargeted dispatch, generator-before-capture sequencing, blanket fallback
Componentreal selector + graph, stubbed generator output → only the affected artifact
Integrationreal follow-up flow, empty capture → merge-diff selection stands
Backend unit suitepasses

Live validation (full battery, demo stack, SELECTIVE_EXECUTION_AFTER_MERGE=true)

Driven over the API against a live demo stack, asserting rendered artifact content (not workflow calls):

#ScenarioVerdictObservation
1Single-member narrowing✅diff=1 → rendered 1/3, others untouched
2Multi-member✅diff=2 → rendered 2/3, third untouched
4Generator → artifact cascade✅after-merge generator writes description, artifact re-renders
5Wide signal (query changed)✅regenerate_all_members=True, rendered 4/4
6Arista mtu (demo-edge)✅startup-config updated to new mtu; oc-interfaces correctly excluded (does not read mtu)
9Diverged main (main gains a member)✅branch member refreshed, no fallback
—Empty diff✅0 generator / 0 artifact — no wasted work
7Composition cascade (artifact inlines another)🔴 gapcomposite permanently inlines the stale section

A/B performance (demo stack, 250-member group, people-only)

Dataset: 250 TestingPerson in a single people group; each member feeds 1 generator
(person_description_generator, writes description) and 3 artifacts — Section config,
Composite config (inlines the section), and the generator-fed Person cascade — so a full
regeneration touches ~250 generators + ~750 artifacts.

Rename one member and merge; wall-clock from merge to regeneration drain, median of 3 reps:

ModeMedianRegen flows dispatched
OFF (full regeneration)~302 s~1001 (250 generators + 750 artifacts)
ON (selective)~27 s~6 (the single impacted member)

~11× faster wall-clock — a conservative floor, since the ON time is dominated by the drain-detection
window; the dispatched regeneration work itself drops ~167× (1001 → 6 flows).

Real-git validation (served bare remote in the demo stack)

Pushed a change to a served bare repository, let the sync import it, then merged — asserting on rendered
artifact content. Each push advanced the repository commit and the definition fingerprint, and the merge
ran the chain against the new code; no CommitNotFoundError, no fallback.

Change in the git repoResult after push → sync → merge
Generator Python code (v1-→v2-)generator runs the new code; artifact shows v2-…
Jinja2 artifact template (Code:→CODE2:)artifact renders with the new template
New artifact definition in .infrahub.ymldefinition imported and selected on the next merge

Exercises the repository commit-resolution path end-to-end (the CommitNotFoundError in the earlier A/B
report was a test-setup artifact — a non-bare remote committed in place — not a product issue).

Closure-completeness gating (served bare remote in the demo stack)

Pushed successive commits to a served bare repo, let the sync import each, then read the definition's
dependencies_complete / fingerprint (and, for the escalation cases, what a merge actually
regenerated). Each row is also covered by the closure-builder unit tests in
backend/tests/unit/git/closure_builder/.

Change in the git repoResult after sync/merge
Jinja2 dynamic {% include var %}transform dependencies_complete=false, fingerprint moved
watch: {files: [...]} declaredclosure trusted again (dependencies_complete=true), watched directory pulled into the dependency list
Broken transform (missing template) beside a healthy onebroken → false, sibling stays true, repository import not aborted
Static {% include "templates/partial.j2" %}included file resolved into the dependency closure, true
Read-only repositoryclosure and fingerprint computed identically to a standard repository
Definition with no computed fingerprint (pre-feature state)its whole repository is force-regenerated on the next merge — even one that changes none of its data — and self-heals once the repository is re-imported
One repository's code changedonly that definition's fingerprint moves; other repositories' fingerprints stay put

How to review

  1. backend/infrahub/core/merge/selective_regen/generator_diff_capturer.py
  2. select_artifacts in selective_regen/orchestrator.py
  3. regeneration_dispatcher.py — _dispatch_plan + _artifacts_from_generator_output
  4. _build_post_merge_regeneration_dispatcher in core/branch/tasks.py
  5. Tests — unit → component → integration
  6. (Optional) run live: add a group member, merge, watch for Targeted N artifact definition(s)

🤖 Generated with Claude Code

https://claude.ai/code/session_012crFmdtPR3Qm2ZCDK8e2Wk