feat(merge): selective post-merge generator & artifact regeneration (IFC-2704)
Why
When a branch merges, the follow-up re-ran every Generator and regenerated every artifact
for every member, regardless of what the merge actually changed — on a real dataset that
spawned thousands of tasks.
Goal: narrow post-merge regeneration to the definitions and members the merge actually
affected, without ever under-executing.
Non-goals: Jinja2 computed attributes, display labels and HFIDs keep their separate coalesced
path (merge-recompute.md); Python-transform computed attributes and profile refresh are out of
scope.
Closes IFC-2704
What changed
Behavioral
- New
selective_execution_after_mergesetting (INFRAHUB_SELECTIVE_EXECUTION_AFTER_MERGE),
enabled by default. A merge captures its diff and dispatches only definitions whose query,
definition, or code inputs changed — and only for affected members. - Guiding rule: over-execution is acceptable, under-execution is not. Every uncertain signal
(flag off, missing/unreadable diff summary, no computed fingerprint, incomplete dependency
closure) falls back to full regeneration. - On a direct merge that runs a Generator, artifacts consuming that Generator's output are
regenerated after it completes. - Fix: regeneration no longer aborts when a definition references a repository with no changes in
the branch diff — the missing repository is skipped. - Fix: Generator instances are narrowed to impacted members instead of rerunning every instance in
the group.
Implementation
core/regeneration/— shared selection primitives (definitions, impact, members, predicates).core/merge/selective_regen/— selection orchestrator, per-type definition selectors, gate,
fallbacks, generator diff capturer.core/merge/regeneration_dispatcher.py—PostMergeRegenerationDispatcher.core/diff/summary_cache.py/summary_serializer.py— serialize the in-memory enriched diff
before it is frozen, cache it under the merge key.proposed_change/tasks.pynet −534 lines as logic moved into the shared modules.
What stayed the same
- No schema changes, no migrations, no GraphQL schema changes.
GRAPH_VERSIONuntouched at 74. - Disabling the flag restores the blanket path byte-for-byte.
How to review
This branch is the aggregate of seven PRs that were each reviewed and merged into it. All seven
were individually approved, so the assembled result is best read as a whole rather than
line-by-line. The table is also the suggested review order.
Work added on top of that sum, in response to review feedback on this PR, is listed in
Commits added for review below and is the part that has not been reviewed yet.
| # | PR | Summary |
|---|---|---|
| 1 | #9886 docs(specs) | Design documents — spec, plan, research, data model, contracts, and the performance-validation method. Start here for intent. |
| 2 | #9924 feat(merge) | The core feature — selection engine (gate, selectors, orchestrator), diff serializer/cache, dispatcher, and the selective_execution_after_merge flag. |
| 3 | #9928 refactor(regeneration) | Extracts the shared selection primitives into core/regeneration/ so the merge and proposed-change paths stop duplicating them. |
| 4 | #9971 fix(regeneration) | Corrects the exception catch and subscriber member resolution surfaced by the refactor. |
| 5 | #10028 feat(regeneration) | Safe-fallback escalation and the post-merge generator cascade — the fallback paths where under-execution would hide. |
| 6 | #10033 feat(regeneration) | Targets artifacts from an after-merge generator's own output, so a generator-driven change propagates to the artifacts consuming it. |
| 7 | #10037 docs(regeneration) | Wrap-up — reconciles the IFC-2704 artifacts, knowledge reference, and QA checklist against what shipped. |
For a narrative of the assembled flow, read
dev/knowledge/backend/selective-merge-regeneration.md.
Commits added for review
The commits pushed on top of the seven merged PRs answer review feedback raised on this PR. They
are not covered by the approvals above and are the part still needing review — two of them
change behaviour, the rest are tests and documentation. git log origin/develop.. lists them, and
each commit message states what it changes and why.
How to test
uv run invoke backend.test-unit
uv run pytest backend/tests/component/proposed_change/test_merge_selective_regen.py
uv run pytest backend/tests/integration/proposed_change/test_merge_selective_regen.py
Local pre-CI on the rebased branch is green: format, ruff/ty/mypy (1564 files), Biome (1500
files), Betterer (201 issues, unchanged), docs lint + vale, all generated-file and generated-doc
validation, and 2012 unit tests passing.
Explicit validation scenarios live in the intermediate PR descriptions rather than being repeated
here:
| PR | Section | What it documents |
|---|---|---|
| #9924 | How to test | Same-build A/B on infra demo + infrahub-demo-edge (30 devices, 2 artifact defs, 4 generator defs). Four scenarios — single-kind change, no-op read, membership-only, direct-merge generator cascade — flag-off 88 regen flows versus 24/0/3/3 flag-on, zero under-execution. |
| #10028 | Working scenarios | Seven-row matrix: generator→artifact cascade on both merge kinds, generator opt-out, updated transform on main, new definition on main, diverged main, member narrowing, and the one known gap (content composition refresh). |
| #10033 | How tested + Live validation | Full battery driven over the API against a live demo stack asserting rendered artifact content: single/multi-member narrowing, generator cascade, wide query signal, Arista mtu on demo-edge, diverged main, empty diff. Plus a 250-member A/B perf run. |
Impact & rollout
- Backward compatibility: no breaking changes; setting the flag to
falserestores the exact
prior full-regeneration behavior. The no-under-execution guarantee holds either way via the
full-regeneration fallback. - Performance: the point of the change — dispatched-task count for a small merge drops from
proportional-to-instance-size to proportional-to-affected-set (73–100% fewer regen flows across
the validated scenarios in #9924). - Known limitation: a composite artifact that inlines an upstream artifact is not refreshed
after a merge that changes the upstream. Documented and out of scope — see the composition
cascade rows in #10028 and #10033. - Config/env changes:
INFRAHUB_SELECTIVE_EXECUTION_AFTER_MERGE, defaulttrue
(docker-compose and configuration reference regenerated). - Deployment notes: safe to deploy; the selective path is active by default and revertible via
the flag.
Checklist
- Tests added/updated
- Changelog entry added (3 fragments)
- External docs updated (
docs/docs/reference/configuration.mdx) - Internal .md docs updated (knowledge + spec docs)
- I have reviewed AI generated content