perf(computed-attribute): batch Python transform recompute
Why
When a change touches data that Python-transform computed attributes read, every affected node was recomputed through its own full path: a git repository init per node, one GraphQL update mutation per node, and every mutation's events re-triggered target queries and further recompute dispatches. One device-type rename could keep an instance churning for minutes, and at scale it takes the instance down. This is the "merge leaves the instance unusable" pain reported by large deployments.
Goal: absorb a wide fan-out as one bounded pass per chunk, with the same final values and the same events for real changes.
Non-goals: scoping the fan-out itself (every node of the kind vs only the affected ones) and a recovery surface for failed nodes, both are follow-ups.
The improvements
The scenario for all of them is a Python computed attribute whose source data changes with many reader nodes, most visibly a branch merge or rebase, but the same path fires on a direct data change or a transform update that fans out to a whole kind.
- One git repository init per batch instead of one per node. The recompute flow receives a chunk of node ids (up to ~250) and initializes the transform's repository once, before this every single node paid its own repository init.
- One bulk write instead of one GraphQL mutation per node. Values persist through the shared bulk recompute writer in bounded transactions (100 nodes per transaction), no per-node
InfrahubUpdateComputedAttributemutation, no per-node client round-trip. - Unchanged values emit no events, which removes the echo storm. Before, every per-node mutation emitted
NodeUpdatedevents even when the recomputed value was identical to the stored one, and those events re-triggered thequery-computed-attribute-transform-targetsandgraphql-query-group-updateautomations, which dispatched further recomputes, a self-sustaining storm. Now a node whose save produces no effective change emits nothing and dispatches nothing, so a wave of recomputes that mostly lands on already-correct values simply stops. - One bad node no longer hurts its siblings. A transform that raises or returns a non-string for one node is skipped with its previous value intact and a logged reason, the rest of the batch persists, and the flow ends with a
submitted/written/skippedsummary line.
Plus one visibility fix: the branch tag is set on each submission at creation so the recompute runs stay visible in branch-filtered task views, tags added mid-run can be dropped by later in-flow tag updates.
How we know it is an improvement
Measured, not inferred. The TestMergeRecomputePython suite in infrahub-private-tests restores the same dataset backup, renames a device type on a branch, merges, waits until every affected device is recomputed, and measures how long the instance keeps churning afterwards and how many flow runs the merge caused. We ran it twice on develop_x-small, once with develop (baseline run) and once with this branch (run with this PR), same dataset, same scenario, the only difference between the two runs is the commits in this PR.
| Metric (merge) | develop | this branch |
|---|---|---|
| Trailing recompute churn after merge | 275.2 s | ~1 s |
query-computed-attribute-transform-targets runs | 1,321 | 1 |
| Process flow runs | 169 | 1 |
| Client-visible update mutations | 1 per node | 0 |
At scale develop does not degrade gracefully, it collapses: on develop_large, the same scenario on develop spawned 73,149 flow runs in 110 minutes (99% dispatch echo, 53k group updates plus 19k target queries, near-zero actual transform work) and died with Neo4j bolt-thread and file-descriptor exhaustion. The same scenario with this branch, on a deliberately smaller database because the shared CI hosts kept OOM-killing full-size setups, survived without taking the host down. The churn that remains there is the reverse-index bookkeeping from dispatching to every node of the kind, which this PR deliberately does not touch, so the at-scale data confirms fan-out scoping as the necessary follow-up rather than a nice-to-have.
How we know nothing broke
- The perf suite asserts correctness, not just speed: after the merge, every affected device's computed value matches the expected new value, on both runs above.
- The existing functional computed-attribute suite (real git repository, real transforms, end to end) runs unchanged and passes.
- The write path is not new code, it is the same bulk writer the Jinja2 computed attributes already persist through.
- The behavior contract is preserved: same final values, same per-node
NodeUpdatedevents for real changes, the publicInfrahubUpdateComputedAttributemutation stays (just no longer used internally), and the flow parameters are unchanged so the Prefect deployment contract is stable. - New tests pin the new behavior: unit tests for the partition/skip logic, a functional test asserting an oversized fan-out splits into chunks with every id exactly once and the branch tag on every submission, and a component test characterizing today's whole-kind fan-out so the future scoping work has a baseline to assert against.
Built on the Jinja2 work, and what Python transforms still cannot do
This deliberately reuses the bulk recompute machinery Guillaume built for the Jinja2 computed attributes (BulkRecomputeDispatcher/BulkRecomputeWriter), no second write path, and it is verified with his TestMergeRecomputePython suite from infrahub-private-tests #29.
The part we cannot inherit: Jinja2 values are template renders the backend computes directly, so on merge they are recomputed in one coalesced in-process pass. Python transforms execute user code from a git repository on the task workers, they cannot join that pass, so they remain event-dispatched flows and a merge still dispatches a recompute for every node of the kind rather than only the changed ones. This PR makes that dispatch cheap and non-amplifying, scoping it to the affected nodes is the next big lever (see next steps).
Suggested review order
backend/infrahub/computed_attribute/tasks.py, theprocess_transformrewrite plus the tagged submission sites.- Tests: unit (partition/skip, submitter tags), functional (chunked submission and tags), component (fan-out characterization).
- The
add_tagscommit pair (de6664dedreverted by58680ba2e) documents a rejected fix, kept for history, the final approach tags at submission instead.
The design artifacts (spec, plan, critique, tasks) live in dev/specs/004-batch-python-recompute/.
How to test
uv run pytest backend/tests/unit/computed_attribute/ -q
uv run pytest backend/tests/functional/computed_attributes/ -q
At-scale validation runs through infrahub-private-tests test-dataset.yml with test_filter=TestMergeRecomputePython, links above.
Impact & rollout
- Backward compatibility: no schema changes, no API changes, event contract unchanged for real value changes.
- Performance: see above, the trailing recompute after a wide-impact change drops from minutes to about a second on the reference dataset.
- Deployment notes: safe to deploy, rollback is a clean revert, no migration.
Next steps
-
Handle failed/skipped nodes more gracefully, likely raise after the dispatch so the flow run shows failed while healthy siblings stay persisted:
await dispatcher.dispatch(writes=writes, ...) # healthy siblings already persisted if skipped: raise RecomputeError(f"{len(skipped)} node(s) failed: {skipped}") -
A queryable recompute-failure surface with a one-click re-run (the
RecomputeComputedAttributemutation is the existing lever). -
Scope the merge fan-out to affected nodes instead of every node of the kind, that is the next big lever for giant-branch merges and the one thing the Jinja2 path also still needs.
-
Phase-level spans/counters (repo-init, read, execute, write) to restore the per-node observability the batch path removed.
Checklist
- Tests added/updated
- Changelog entry added
- External docs updated (not user-facing, behavior-preserving)
- Internal .md docs updated (
dev/knowledge/backend/computed-attributes.md) - I have reviewed AI generated content
๐ค Generated with Claude Code