feat(merge): selective post-merge generator & artifact regeneration

Merged
PO
4mo ago
+2664-134

Why

Merging a branch currently re-runs every generator and regenerates every artifact,
regardless of what the merge actually changed. On a large dataset — the originating report
(IFC-2306) cites ~4,000 artifacts across ~40 branches — a single merge spawns thousands of
artifact-generation tasks and leaves the API and UI effectively unusable for 10–15 minutes.

Goal: make post-merge execution selective — run only the generators and regenerate only the
artifacts whose inputs the merge diff affected, scoped to the affected group members.

Guiding invariant: over-execution is acceptable, under-execution is not. Whenever the affected
set cannot be determined with confidence, the merge falls back to regenerating everything rather
than risk leaving an artifact stale.

Non-goals: no change to any regeneration caller outside the merge follow-up (proposed-change
pipeline, manual regeneration are untouched); the shared-primitive refactor is deferred (see below).

Closes IFC-2893, IFC-2894, IFC-2895, IFC-2898

Scope within IFC-2704

Where this PR sits in the parent-feature breakdown:

ChildSummaryThis PR?
IFC-2893Setup✅ closes (already listed)
IFC-2894Foundational (blocking prereqs)✅ closes (already listed)
IFC-2895MVP — selective regen on merge✅ closes (already listed)
IFC-2898Reversible rollout via configuration✅ this is the config flag — closes
IFC-2908internal: refactoring into regeneration package🔜 referenced as follow-up (Phase 1)
IFC-2911internal: remove SDK usage from selective regen🔜 follow-up (Phase 2 of same doc)
IFC-2896Safe fallback to full regeneration⚠️ mechanism is in this PR, story is a later increment
IFC-2897Repository code changes still regenerate❌ later increment (US2)
IFC-2899Direct-merge generator cascade (D7)⚠️ only validated here (test #8), not closed
IFC-2900 / 2907Polish / additional thoughts❌ later

What changed

Behavioral changes

  • On merge, only definitions whose inputs are in the merge diff are
    dispatched, narrowed to the affected members.
  • Applies identically to direct branch merges and merges through a proposed change.
  • New config flag selective_execution_after_merge gates the behavior.
  • Comprehensive fallback to full regeneration on any uncertainty: feature disabled, no diff summary
    captured, an unloadable summary, or any failure during selection/dispatch.

Implementation notes

  • The enriched branch diff is captured in the merge orchestrator, serialized into the NodeDiff summary the selection predicates consume, and cached keyed by the freeze-stable diff-root uuid. The follow-up loads the summary by key.
  • A new selective_regen/ package builds the regeneration plan from the diff summary.
  • Selection reuses the proposed-change regeneration-trigger primitives rather than reimplementing
    them.

What stayed the same: no schema changes; no change to any regeneration caller outside the two
post-merge trigger submissions; disabled flag reproduces prior behavior byte-for-byte.

How to review

The commits are clean and ordered to be read top to bottom; each is self-contained.

  1. feat(config): add selective_execution_after_merge flag — the on/off surface and default,
    plus the regenerated artifacts (docker-compose, configuration reference, OpenAPI + frontend REST
    types). Start here.
  2. feat(merge): add merge diff serializer/cache components and share PC primitives — how the
    enriched diff becomes the NodeDiff summary and is cached, and the proposed-change primitives
    that become shared. Foundational; no merge behavior wired yet.
  3. feat(merge): add selective post-merge regeneration selection engine — the selective_regen/
    package (orchestrator → gate → selectors → impact resolver). The core logic; the dependency
    graph above maps this commit.
  4. feat(merge): dispatch and thread the merge diff summary into the follow-up — the wiring:
    diff capture in the merge orchestrator, the dispatcher's enable/fallback decision tree, and the
    composition root in branch/tasks.py. The correctness surface for the no-under-execution invariant.
  5. test(merge): selective post-merge regeneration — component-tier selection coverage of the
    selective dispatch (recorder-based, against a real graph) plus the engine unit suites. Real
    regenerated-output is deferred to an API-tier test — the integration testcontainer harness
    cannot execute the render flow's worker→server callback.
  6. docs(specs): record IFC-2704 US1 progress, D7 spike, and interim default — spec/plan
    context; skim.
  7. chore(config): regenerate docker-compose.yml — generated file; mechanical.
  8. docs(specs): reconcile IFC-2704 test tiers and perf-validation status — aligns the
    spec/plan/tasks with the shipped test tiers and the perf-validation results; skim.

Selective regeneration package

COMPONENT / DEPENDENCY GRAPH
============================

core/branch/tasks.py :: _build_post_merge_regeneration_dispatcher(log)   [composition root]
        |   wires everything per merge follow-up
        v
+------------------------------------------------------------------+
| PostMergeRegenerationDispatcher      (regeneration_dispatcher.py) |
|   inject: workflow . selector . summary_cache . log              |
|   dispatch():                                                    |
|     flag off / no cache key / cache miss / error -> blanket      |
|                                    (TRIGGER_* every definition)  |
|     else -> selector.build_plan() -> submit only what it returns |
+------------------------------------------------------------------+
        |  selector : RegenerationSelector (Protocol)
        v
+------------------------------------------------------------------+
| MergeSelectiveRegeneration                     (orchestrator.py) |
|   built by build_merge_selective_regeneration(client, log)       |
|   modified_kinds = get_modified_kinds(diff)                      |
|   generator_selector.select() + artifact_selector.select()      |
|   -> SelectiveRegenerationPlan { generator_runs, artifact_gens } |
+------------------------------------------------------------------+
        |
        v
+------------------------------------------------------------------+
| DefinitionSelectorBase[Def, Req]   (definition_selector/base.py) |
|   ABC / template method                                          |
|   inject: client . gate . impacted_resolver . log                |
|   per definition:                                                |
|     1. gate.evaluate()            -> selected? / all-members?    |
|     2. impacted_resolver.resolve()-> which members changed       |
|     3. _narrow_members_filter()   -> subset, or empty = "all"    |
|     4. _build_request()           -> skip if no members render   |
+------------------------------------------------------------------+
     |  abstract hooks: _load_definitions . _fetch_group . _should_render . _build_request
     +-- GeneratorSelector   generators flagged execute_after_merge
     +-- ArtifactSelector    artifact definitions

  shared, injected into BOTH selectors:
    - DefinitionGate (gate.py)                 -> GateResult { managed_branch, selected }
        selects on: query changed | definition/fingerprint changed |
                    modified kind read by query | target group touched (membership-only)
    - ImpactedSubscriberResolver (impacted.py)
        field-level impact -> ALL (widen to all members) | NONE (skip) | specific ids

  Guiding invariant: every uncertain signal WIDENS the selection; it never narrows below
  the set of members whose inputs changed, so nothing affected is left stale.

Follow-up refactor (uniformization, deferred)

The selective-regeneration path currently reuses the proposed-change selection primitives by
importing underscored internals from proposed_change/tasks.py, which forces a lazy import in
branch/tasks.py to avoid an import cycle. A separate refactor will uniformize the two paths by
extracting the shared cluster into a dependency-free leaf package (core/regeneration/), promoting
the shared names to public API and breaking the cycle (Phase 1, IFC-2908), then moving the merge
path off the SDK client to direct database access (Phase 2, IFC-2911). Both are behavior-neutral and
intentionally not part of this increment.

How to test

Automated coverage: unit suites for the selection engine (gate, selectors, orchestrator,
diff serializer/cache, dispatcher) plus a component-tier selection test driving the dispatcher
against a real graph (backend/tests/component/proposed_change/test_merge_selective_regen.py).

Manual A/B validation on a real dataset:

Environment / build

  • Dataset: infra demo + infrahub-demo-edge (read-only). 30 devices; 2 artifact defs
    (20 artifacts); 4 generator defs (all execute_after_merge). Baseline blanket = 80 leaf
    regen flows (40 query-group + 10 openconfig + 10 startup-config + 20 generator-member).
  • Flag: INFRAHUB_SELECTIVE_EXECUTION_AFTER_MERGE. Same-build A/B: off, then on.
  • Metric: total regeneration-family Prefect flow runs in the drained follow-up window
    (leaf flows + orchestration wrappers), per the plan's step-5 counting recipe.

Results

Headline = total regen-family flows; flag-off is scenario-independent = 88.

#ScenarioChangeFlag offFlag onUnder-exec
1Single-kind changeInfraInterfaceL3.description, 1 arista device88240
2No-op readBuiltinTag.description (no def reads it)8800
4Membership-onlyadd existing core device to arista group8830
8Direct-merge gen. cascadeInfraCircuit.vendor_id (generators only)8830

Impact & rollout

  • Backward compatibility: selective execution is on by default; set
    INFRAHUB_SELECTIVE_EXECUTION_AFTER_MERGE=false to restore the exact prior full-regeneration
    behavior. The no-under-execution guarantee holds either way via the full-regeneration fallback.
  • Performance: the point of the change — dispatched-task count for a small merge drops from
    proportional-to-instance-size to proportional-to-affected-set (73–100% fewer regen flows across
    the validated scenarios).
  • Config/env changes: new INFRAHUB_SELECTIVE_EXECUTION_AFTER_MERGE flag, default true
    (docker-compose and configuration reference regenerated).
  • Deployment notes: safe to deploy; selective path is active by default and revertible via the flag.