feat(regeneration): target artifacts from an after-merge generator's output
Summary
Refines the post-merge generator → artifact cascade: instead of regenerating every artifact once
a generator runs, regenerate only the artifacts its output affects.
Change
A generator dispatched by the follow-up runs after the merge diff was captured, so its writes are
absent from that diff. Once every generator has run, this diffs the target branch over the window
they ran in and selects only the artifacts that read the changed data (reusing the artifact half of
the merge selector). Full regeneration stays the fallback if that output can't be captured or
selected, so no artifact is left stale. Direct and proposed-change merges take the same path.
How tested
| Level | Coverage |
|---|---|
| Unit | targeted dispatch, generator-before-capture sequencing, blanket fallback |
| Component | real selector + graph, stubbed generator output → only the affected artifact |
| Integration | real follow-up flow, empty capture → merge-diff selection stands |
| Backend unit suite | passes |
Live validation (full battery, demo stack, SELECTIVE_EXECUTION_AFTER_MERGE=true)
Driven over the API against a live demo stack, asserting rendered artifact content (not workflow calls):
| # | Scenario | Verdict | Observation |
|---|---|---|---|
| 1 | Single-member narrowing | ✅ | diff=1 → rendered 1/3, others untouched |
| 2 | Multi-member | ✅ | diff=2 → rendered 2/3, third untouched |
| 4 | Generator → artifact cascade | ✅ | after-merge generator writes description, artifact re-renders |
| 5 | Wide signal (query changed) | ✅ | regenerate_all_members=True, rendered 4/4 |
| 6 | Arista mtu (demo-edge) | ✅ | startup-config updated to new mtu; oc-interfaces correctly excluded (does not read mtu) |
| 9 | Diverged main (main gains a member) | ✅ | branch member refreshed, no fallback |
| — | Empty diff | ✅ | 0 generator / 0 artifact — no wasted work |
| 7 | Composition cascade (artifact inlines another) | 🔴 gap | composite permanently inlines the stale section |
A/B performance (demo stack, 250-member group, people-only)
Dataset: 250 TestingPerson in a single people group; each member feeds 1 generator
(person_description_generator, writes description) and 3 artifacts — Section config,
Composite config (inlines the section), and the generator-fed Person cascade — so a full
regeneration touches ~250 generators + ~750 artifacts.
Rename one member and merge; wall-clock from merge to regeneration drain, median of 3 reps:
| Mode | Median | Regen flows dispatched |
|---|---|---|
| OFF (full regeneration) | ~302 s | ~1001 (250 generators + 750 artifacts) |
| ON (selective) | ~27 s | ~6 (the single impacted member) |
~11× faster wall-clock — a conservative floor, since the ON time is dominated by the drain-detection
window; the dispatched regeneration work itself drops ~167× (1001 → 6 flows).
Real-git validation (served bare remote in the demo stack)
Pushed a change to a served bare repository, let the sync import it, then merged — asserting on rendered
artifact content. Each push advanced the repository commit and the definition fingerprint, and the merge
ran the chain against the new code; no CommitNotFoundError, no fallback.
| Change in the git repo | Result after push → sync → merge |
|---|---|
Generator Python code (v1-→v2-) | generator runs the new code; artifact shows v2-… |
Jinja2 artifact template (Code:→CODE2:) | artifact renders with the new template |
New artifact definition in .infrahub.yml | definition imported and selected on the next merge |
Exercises the repository commit-resolution path end-to-end (the CommitNotFoundError in the earlier A/B
report was a test-setup artifact — a non-bare remote committed in place — not a product issue).
Closure-completeness gating (served bare remote in the demo stack)
Pushed successive commits to a served bare repo, let the sync import each, then read the definition's
dependencies_complete / fingerprint (and, for the escalation cases, what a merge actually
regenerated). Each row is also covered by the closure-builder unit tests in
backend/tests/unit/git/closure_builder/.
| Change in the git repo | Result after sync/merge |
|---|---|
Jinja2 dynamic {% include var %} | transform dependencies_complete=false, fingerprint moved |
watch: {files: [...]} declared | closure trusted again (dependencies_complete=true), watched directory pulled into the dependency list |
| Broken transform (missing template) beside a healthy one | broken → false, sibling stays true, repository import not aborted |
Static {% include "templates/partial.j2" %} | included file resolved into the dependency closure, true |
| Read-only repository | closure and fingerprint computed identically to a standard repository |
| Definition with no computed fingerprint (pre-feature state) | its whole repository is force-regenerated on the next merge — even one that changes none of its data — and self-heals once the repository is re-imported |
| One repository's code changed | only that definition's fingerprint moves; other repositories' fingerprints stay put |
How to review
backend/infrahub/core/merge/selective_regen/generator_diff_capturer.pyselect_artifactsinselective_regen/orchestrator.pyregeneration_dispatcher.py—_dispatch_plan+_artifacts_from_generator_output_build_post_merge_regeneration_dispatcherincore/branch/tasks.py- Tests — unit → component → integration
- (Optional) run live: add a group member, merge, watch for
Targeted N artifact definition(s)
🤖 Generated with Claude Code