feat(merge): selective post-merge generator & artifact regeneration
Why
Merging a branch currently re-runs every generator and regenerates every artifact,
regardless of what the merge actually changed. On a large dataset — the originating report
(IFC-2306) cites ~4,000 artifacts across ~40 branches — a single merge spawns thousands of
artifact-generation tasks and leaves the API and UI effectively unusable for 10–15 minutes.
Goal: make post-merge execution selective — run only the generators and regenerate only the
artifacts whose inputs the merge diff affected, scoped to the affected group members.
Guiding invariant: over-execution is acceptable, under-execution is not. Whenever the affected
set cannot be determined with confidence, the merge falls back to regenerating everything rather
than risk leaving an artifact stale.
Non-goals: no change to any regeneration caller outside the merge follow-up (proposed-change
pipeline, manual regeneration are untouched); the shared-primitive refactor is deferred (see below).
Closes IFC-2893, IFC-2894, IFC-2895, IFC-2898
Scope within IFC-2704
Where this PR sits in the parent-feature breakdown:
| Child | Summary | This PR? |
|---|---|---|
| IFC-2893 | Setup | ✅ closes (already listed) |
| IFC-2894 | Foundational (blocking prereqs) | ✅ closes (already listed) |
| IFC-2895 | MVP — selective regen on merge | ✅ closes (already listed) |
| IFC-2898 | Reversible rollout via configuration | ✅ this is the config flag — closes |
| IFC-2908 | internal: refactoring into regeneration package | 🔜 referenced as follow-up (Phase 1) |
| IFC-2911 | internal: remove SDK usage from selective regen | 🔜 follow-up (Phase 2 of same doc) |
| IFC-2896 | Safe fallback to full regeneration | ⚠️ mechanism is in this PR, story is a later increment |
| IFC-2897 | Repository code changes still regenerate | ❌ later increment (US2) |
| IFC-2899 | Direct-merge generator cascade (D7) | ⚠️ only validated here (test #8), not closed |
| IFC-2900 / 2907 | Polish / additional thoughts | ❌ later |
What changed
Behavioral changes
- On merge, only definitions whose inputs are in the merge diff are
dispatched, narrowed to the affected members. - Applies identically to direct branch merges and merges through a proposed change.
- New config flag
selective_execution_after_mergegates the behavior. - Comprehensive fallback to full regeneration on any uncertainty: feature disabled, no diff summary
captured, an unloadable summary, or any failure during selection/dispatch.
Implementation notes
- The enriched branch diff is captured in the merge orchestrator, serialized into the
NodeDiffsummary the selection predicates consume, and cached keyed by the freeze-stable diff-root uuid. The follow-up loads the summary by key. - A new
selective_regen/package builds the regeneration plan from the diff summary. - Selection reuses the proposed-change regeneration-trigger primitives rather than reimplementing
them.
What stayed the same: no schema changes; no change to any regeneration caller outside the two
post-merge trigger submissions; disabled flag reproduces prior behavior byte-for-byte.
How to review
The commits are clean and ordered to be read top to bottom; each is self-contained.
feat(config): add selective_execution_after_merge flag— the on/off surface and default,
plus the regenerated artifacts (docker-compose, configuration reference, OpenAPI + frontend REST
types). Start here.feat(merge): add merge diff serializer/cache components and share PC primitives— how the
enriched diff becomes theNodeDiffsummary and is cached, and the proposed-change primitives
that become shared. Foundational; no merge behavior wired yet.feat(merge): add selective post-merge regeneration selection engine— theselective_regen/
package (orchestrator → gate → selectors → impact resolver). The core logic; the dependency
graph above maps this commit.feat(merge): dispatch and thread the merge diff summary into the follow-up— the wiring:
diff capture in the merge orchestrator, the dispatcher's enable/fallback decision tree, and the
composition root inbranch/tasks.py. The correctness surface for the no-under-execution invariant.test(merge): selective post-merge regeneration— component-tier selection coverage of the
selective dispatch (recorder-based, against a real graph) plus the engine unit suites. Real
regenerated-output is deferred to an API-tier test — the integration testcontainer harness
cannot execute the render flow's worker→server callback.docs(specs): record IFC-2704 US1 progress, D7 spike, and interim default— spec/plan
context; skim.chore(config): regenerate docker-compose.yml— generated file; mechanical.docs(specs): reconcile IFC-2704 test tiers and perf-validation status— aligns the
spec/plan/tasks with the shipped test tiers and the perf-validation results; skim.
Selective regeneration package
COMPONENT / DEPENDENCY GRAPH
============================
core/branch/tasks.py :: _build_post_merge_regeneration_dispatcher(log) [composition root]
| wires everything per merge follow-up
v
+------------------------------------------------------------------+
| PostMergeRegenerationDispatcher (regeneration_dispatcher.py) |
| inject: workflow . selector . summary_cache . log |
| dispatch(): |
| flag off / no cache key / cache miss / error -> blanket |
| (TRIGGER_* every definition) |
| else -> selector.build_plan() -> submit only what it returns |
+------------------------------------------------------------------+
| selector : RegenerationSelector (Protocol)
v
+------------------------------------------------------------------+
| MergeSelectiveRegeneration (orchestrator.py) |
| built by build_merge_selective_regeneration(client, log) |
| modified_kinds = get_modified_kinds(diff) |
| generator_selector.select() + artifact_selector.select() |
| -> SelectiveRegenerationPlan { generator_runs, artifact_gens } |
+------------------------------------------------------------------+
|
v
+------------------------------------------------------------------+
| DefinitionSelectorBase[Def, Req] (definition_selector/base.py) |
| ABC / template method |
| inject: client . gate . impacted_resolver . log |
| per definition: |
| 1. gate.evaluate() -> selected? / all-members? |
| 2. impacted_resolver.resolve()-> which members changed |
| 3. _narrow_members_filter() -> subset, or empty = "all" |
| 4. _build_request() -> skip if no members render |
+------------------------------------------------------------------+
| abstract hooks: _load_definitions . _fetch_group . _should_render . _build_request
+-- GeneratorSelector generators flagged execute_after_merge
+-- ArtifactSelector artifact definitions
shared, injected into BOTH selectors:
- DefinitionGate (gate.py) -> GateResult { managed_branch, selected }
selects on: query changed | definition/fingerprint changed |
modified kind read by query | target group touched (membership-only)
- ImpactedSubscriberResolver (impacted.py)
field-level impact -> ALL (widen to all members) | NONE (skip) | specific ids
Guiding invariant: every uncertain signal WIDENS the selection; it never narrows below
the set of members whose inputs changed, so nothing affected is left stale.
Follow-up refactor (uniformization, deferred)
The selective-regeneration path currently reuses the proposed-change selection primitives by
importing underscored internals from proposed_change/tasks.py, which forces a lazy import in
branch/tasks.py to avoid an import cycle. A separate refactor will uniformize the two paths by
extracting the shared cluster into a dependency-free leaf package (core/regeneration/), promoting
the shared names to public API and breaking the cycle (Phase 1, IFC-2908), then moving the merge
path off the SDK client to direct database access (Phase 2, IFC-2911). Both are behavior-neutral and
intentionally not part of this increment.
How to test
Automated coverage: unit suites for the selection engine (gate, selectors, orchestrator,
diff serializer/cache, dispatcher) plus a component-tier selection test driving the dispatcher
against a real graph (backend/tests/component/proposed_change/test_merge_selective_regen.py).
Manual A/B validation on a real dataset:
Environment / build
- Dataset: infra demo + infrahub-demo-edge (read-only). 30 devices; 2 artifact defs
(20 artifacts); 4 generator defs (allexecute_after_merge). Baseline blanket = 80 leaf
regen flows (40 query-group + 10 openconfig + 10 startup-config + 20 generator-member). - Flag:
INFRAHUB_SELECTIVE_EXECUTION_AFTER_MERGE. Same-build A/B: off, then on. - Metric: total regeneration-family Prefect flow runs in the drained follow-up window
(leaf flows + orchestration wrappers), per the plan's step-5 counting recipe.
Results
Headline = total regen-family flows; flag-off is scenario-independent = 88.
| # | Scenario | Change | Flag off | Flag on | Under-exec |
|---|---|---|---|---|---|
| 1 | Single-kind change | InfraInterfaceL3.description, 1 arista device | 88 | 24 | 0 |
| 2 | No-op read | BuiltinTag.description (no def reads it) | 88 | 0 | 0 |
| 4 | Membership-only | add existing core device to arista group | 88 | 3 | 0 |
| 8 | Direct-merge gen. cascade | InfraCircuit.vendor_id (generators only) | 88 | 3 | 0 |
Impact & rollout
- Backward compatibility: selective execution is on by default; set
INFRAHUB_SELECTIVE_EXECUTION_AFTER_MERGE=falseto restore the exact prior full-regeneration
behavior. The no-under-execution guarantee holds either way via the full-regeneration fallback. - Performance: the point of the change — dispatched-task count for a small merge drops from
proportional-to-instance-size to proportional-to-affected-set (73–100% fewer regen flows across
the validated scenarios). - Config/env changes: new
INFRAHUB_SELECTIVE_EXECUTION_AFTER_MERGEflag, defaulttrue
(docker-compose and configuration reference regenerated). - Deployment notes: safe to deploy; selective path is active by default and revertible via the flag.