docs(specs): IFC-2704 incremental merge regeneration: design documents

Merged
PO
4mo ago
+2020-1

Summary

Design package (spec → plan → tasks) for making post-merge generator and artifact execution
selective instead of blanket. Tracks IFC-2704 (originating incident IFC-2306).

Today every merge reruns every generator and regenerates every artifact regardless of what
changed, spawning thousands of background tasks and leaving the instance unusable for ~20
minutes. This design makes the follow-up run only the definitions and members whose inputs the
merge actually changed, with every uncertain path falling back to full regeneration so the
"over-execution acceptable, under-execution never" invariant holds.

This PR contains planning artifacts only — no implementation code. It is the reviewable
design for the work; tasks.md is the executable breakdown.

How it works

  • Capture the enriched merge diff in BranchMergeOrchestrator.merge from the already-loaded
    in-memory branch_diff (before the diff freezes), serialized into the proposed-change
    NodeDiff summary shape and tagged with the merge target (destination) branch. The cache
    write happens only after the merge's point of no return, so a rolled-back merge writes nothing.
  • Thread only a cache key (the stable diff-root uuid) through run_follow_ups into
    BRANCH_MERGE_POST_PROCESS, avoiding the Prefect parameter-size problem.
  • Select by reusing the proposed-change definition predicates (_query_changed,
    _definition_changed, MODIFIED_KINDS ∩ query_models, execute_after_merge) plus the
    fingerprint-in-diff signal for repo-code changes, then reconcile members against the live
    group on the target branch
    — mapping impacted subscribers to member ids and force-rendering
    new members (closing the artifact limit trap via a new member-id filter).
  • Fall back to full regeneration on cache miss, capture failure, null fingerprint, or
    incomplete dependency closure; gate the whole behavior behind
    INFRAHUB_SELECTIVE_EXECUTION_AFTER_MERGE (default on) for baseline and rollback.

Artifacts

dev/specs/ifc-2704-incremental-merge-regen/: spec.md, plan.md, research.md (decisions
D1–D9 + D4a), data-model.md, contracts/ (merge-diff-summary, branch-merge-post-process,
artifact-generate-members), quickstart.md, tasks.md (43 tasks), and critiques/ (dual-lens
review — caught and closed the member-level under-execution holes before implementation).

Notes for reviewers

  • Correctness-complete MVP is Foundational + US1 (selective regeneration) + US3 (safe fallback);
    US1 alone can under-execute and is not shippable without US3.
  • The load-bearing safety steps are member reconciliation (T016) and the fallbacks (T024).
  • Two items are flagged as blocking prerequisites for implementation: the direct-merge
    generator→artifact cascade spike (D7/T003) and the null-fingerprint repo-signal verification
    (D6/T025).
  • No core-schema, GraphQL-schema, or new-dependency changes; reuses IFC-2844 fingerprints and
    IFC-2738/INFP-409 dependency-closure fields.

🤖 Generated with Claude Code


Summary by cubic

Design package for IFC-2704 to make post-merge generator and artifact execution selective, running only definitions and members affected by the merge diff. Planning docs only — no implementation changes.

  • New Features

    • Capture the enriched merge diff in the orchestrator, serialize to NodeDiff tagged with the target (destination) branch, write the cache entry after the point of no return under a merge-scoped key with :diff_summary, pass only the cache key to the post-merge workflow; summary-load failures normalize to ResourceNotFoundError.
    • Replace blanket triggers with selection on the target branch: reuse proposed-change predicates, add a group‑membership gate, reconcile impacted subscribers to member ids, use fingerprints to detect repo-code changes, and add a member-id members filter to RequestArtifactDefinitionGenerate to avoid the limit trap.
    • Fail safe: on cache/diff/dependency issues or null fingerprints without a confirmed repo commit, fall back to full regeneration (repo-wide when needed); thread proposed_change_id and use is_proposed_change_merge to drive the direct-merge generator→artifact cascade; gated by INFRAHUB_SELECTIVE_EXECUTION_AFTER_MERGE.
  • Documentation

    • Adds spec, plan, research, data model, contracts, quickstart, tasks, checklists, alignment check, and a live-application performance validation plan with a recorded baseline (~80 regeneration flows) and a turnkey retest recipe; counting runs inside the worker via docker exec, passes the host watermark with -e WM, and reads PREFECT_API_URL from the container env.
    • Extends the perf plan to report a recompute window (min start → max end) over the same filtered regeneration flows and a drained check; time is directional (~78s baseline) and only comparable flag-off vs flag-on on one build — the deterministic flow count remains the headline metric.