Pull requests for INFP-739:

1. #11004: Store the cost level of each stored GraphQL query. Base: #11003 (fac/graphql-query-cost-impl-rrake), which adds the query cost estimate.
2. #11005: Warn about a high-cost stored query in the web UI.
3. Two alternatives, both based on #11005:
   • #11006 (this pull request): Limit or skip the artifact renders of high-cost queries.
   • #11019: Run the database queries of high-cost requests with the pipelined runtime of Neo4j.

Why: high-cost artifact renders slow down the API for every user

When a proposed change or a merge affects the artifacts of a high-cost stored query, Infrahub submits all their renders at once, and each render runs the query. On the test data set described below, three renders for the large network ran at the same time. Meanwhile, a light GraphQL query sent every 5 seconds took up to 2.2 seconds during the checks of the proposed change and up to 4.1 seconds after the merge. The two settings below let an administrator run fewer of these renders at once, or leave them out of the checks of a proposed change.

Not in this pull request:

• An end-to-end test for skipped checks: the end-to-end test stack cannot set server settings.
• Any change for stored queries with the low or unknown level: Infrahub does not estimate their renders, and they never wait.

Two settings, both off by default

• INFRAHUB_GRAPHQL_COST_LIMIT_HIGH_COST_RENDERS: for a stored query with the high level, Infrahub estimates each render with the variables of its target. A render is high-cost when its worst-case database rows are above INFRAHUB_GRAPHQL_COST_HIGH_COST_DATABASE_ROWS, or when it cannot be estimated. At most INFRAHUB_GRAPHQL_COST_HIGH_COST_RENDERS_AT_ONCE high-cost renders (1 by default) run at once across every task worker, in the artifact checks of a proposed change and in artifact generation.
• INFRAHUB_GRAPHQL_COST_SKIP_HIGH_COST_ARTIFACT_CHECKS, which requires the limit: the artifact validator of a proposed change does not render high-cost artifacts. It saves their checks with the new skipped conclusion and a message with the estimate and the limit. A validator with a skipped check and no failed check concludes as skipped, which does not block the merge. After the merge, Infrahub generates these artifacts with the limit.

Skipped checks in the proposed change

• The Checks tab of a proposed change shows a skipped validator and a skipped check with a blue skip icon and the "Skipped" tooltip.
• The chart of each validator kind counts the skipped validators in a blue "Skipped" segment.
• The merge button needs no change: the server decides whether the merge is allowed, and it accepts skipped validators.

How the limit holds back renders

• A high-cost render holds a slot while it runs the query and the Transformation. A slot is a cache key that Infrahub sets only when it does not exist and that expires after one hour, so a slot whose render stopped without releasing it becomes free again.
• A high-cost render writes "Took the high-cost render slot N, after waiting X seconds" and "Released the high-cost render slot N" to the log of its task.
• The flows that start renders run the high-cost ones in as many sequences as there are slots, so renders that wait for a slot do not hold the task workers that light renders need. The artifact generation flow starts the light renders first. The artifact validator starts the sequences at the same time as the light checks.
• The estimate for each target is needed because one stored query can be high-cost for one target and light for others. On the test data set, the cabling plan query is estimated at 991,262 worst-case database rows for the large network and 112,806 for each small network. With the default limit, only the renders for the large network wait. With the stored level alone, all 33 renders would wait.

Results on a test data set

One run for each setting, on Infrahub 1.11.5 with this stack, with one logical network of 938 devices and ten of 80 devices. The probe is a light GraphQL query sent every 5 seconds; its columns show how long the API took to answer it.

Settings	Checks of the proposed change	Probe, 95th percentile / longest	Merge until all artifacts are generated	Probe, 95th percentile / longest	Renders for the large network at once
None	116.5 s	1.9 s / 2.2 s	110.7 s	2.7 s / 4.1 s	3
Limit	239.5 s	0.2 s / 6.0 s	218.9 s	0.02 s / 3.3 s	1
Limit and skip	41.6 s	3.0 s / 3.0 s	234.4 s	0.01 s / 4.6 s	0 in the proposed change, 1 after the merge

• With the limit, the checks and the generation after the merge take about twice as long, and the 95th percentile of the probe is lower.
• Single probe answers still took 3 to 6 seconds. The cause is not known. A possible cause, not verified: the renders for the 30 small networks, which the limit does not hold back.
• With the skip setting, the probe answered only 9 times during the checks of the proposed change.

CPU usage of the stack in more runs

More runs on the same host, with 32 cores, and the same data set: runs 1 and 2 without settings and the limit runs on one day, runs 3 and 4 without settings on the next day. The test read the CPU time of each container from Docker every 5 seconds. CPU usage is in percent of one core (100 % is one core busy for the whole period): the mean over the phase and the peak over 5 seconds. "Whole stack" also includes the cache, the message queue and the other services of the test stack. #11019 compares these runs with its pipelined runtime.

Checks of the proposed change:

Settings	Duration	Probe, 95th percentile / longest	Database, mean / peak	API servers, mean / peak	Task workers, mean / peak	Whole stack, mean / peak
None, run 1	182.7 s *	1.3 s / 11.4 s	71 % / 458 %	254 % / 758 %	12 % / 50 %	370 % / 1,012 %
None, run 2	113.4 s	3.2 s / 4.2 s	103 % / 383 %	339 % / 801 %	14 % / 52 %	496 % / 969 %
None, run 3	118.1 s	5.0 s / 10.7 s	102 % / 410 %	358 % / 801 %	15 % / 59 %	517 % / 1,035 %
None, run 4	116.0 s	1.5 s / 5.4 s	104 % / 426 %	338 % / 756 %	14 % / 56 %	496 % / 957 %
Limit	243.5 s	0.01 s / 4.9 s	49 % / 423 %	163 % / 738 %	8 % / 56 %	249 % / 949 %
Limit and skip	41.3 s	6.0 s / 6.0 s	187 % / 401 %	333 % / 749 %	24 % / 49 %	603 % / 931 %

Merge until all artifacts are generated:

Settings	Duration	Probe, 95th percentile / longest	Database, mean / peak	API servers, mean / peak	Task workers, mean / peak	Whole stack, mean / peak
None, run 1	111.5 s	1.8 s / 5.3 s	57 % / 247 %	379 % / 754 %	11 % / 55 %	483 % / 975 %
None, run 2	112.8 s	0.8 s / 2.3 s	59 % / 251 %	374 % / 799 %	13 % / 51 %	482 % / 1,003 %
None, run 3	118.3 s	0.6 s / 4.2 s	59 % / 320 %	353 % / 779 %	12 % / 54 %	459 % / 995 %
None, run 4	172.3 s *	0.15 s / 3.8 s	39 % / 204 %	242 % / 788 %	9 % / 45 %	320 % / 1,035 %
Limit	237.1 s	0.01 s / 0.2 s	27 % / 273 %	164 % / 700 %	7 % / 50 %	225 % / 969 %
Limit and skip	227.3 s	0.01 s / 1.7 s	28 % / 209 %	168 % / 787 %	7 % / 68 %	233 % / 994 %

* This phase took 55 to 75 s longer than the other phases with the same settings. Run 4 without settings recorded the time of each render: two of the three renders for the large network ran 57 and 74 s longer than the third. The earlier runs did not record these times.

Why some phases take 55 to 75 s longer

• Each render for the large network keeps one API worker process at about 100 % of a CPU core for 80 to 95 s. Each of the two API servers of the test stack runs 4 worker processes, and a worker process runs its Python code on one core at a time.
• When two of the three renders for the large network go to the same worker process, they share its core, and both take about 1.7 times as long. If each render went to one of the 8 processes at random, this would happen in 34 % of the phases. On the second day, 7 of the 20 phases in which every render ran at once took 55 to 75 s longer, counting runs with other settings, which are not in this pull request.
• In two slow phases, the CPU time of each API worker process, read every 5 seconds, showed one process at 80 % of a core or more for 148 s and 153 s, and another for 82 s and 87 s. In a fast phase, three processes were each busy for 87 to 92 s.
• So the duration of one run does not show the effect of a setting. The other phases without settings took 111 to 119 s.
• With the limit, only one render for the large network runs at a time, so two of them cannot share a worker process.

CPU usage with and without the limit

• The API servers used the most CPU in every run: up to 8 cores, against at most 4.6 cores for the database and at most 0.7 core for the task workers.
• With the limit, the whole stack used about the same CPU time as without settings (mean usage multiplied by duration), spread over about twice the duration: 606 CPU seconds against 562 to 676 in the checks of the proposed change, 533 against 539 to 551 after the merge. The mean usage is therefore lower. The peak over 5 seconds of the whole stack stays between 931 % and 1,035 % in every run, with or without the limit.
• With limit and skip, the checks of the proposed change used 249 CPU seconds, because they did not render the artifacts of the large network.
• With limit and skip, the probe answered only 8 times during the checks of the proposed change.

How to review

1. backend/infrahub/graphql/cost/render_cost.py (which renders are high-cost) and backend/infrahub/graphql/cost/render_slots.py (the slots)
2. backend/infrahub/artifacts/high_cost.py (skipped checks and sequences)
3. Where the renders start: backend/infrahub/proposed_change/tasks.py, backend/infrahub/git/tasks.py, backend/infrahub/artifacts/tasks.py
4. The skipped conclusion: backend/infrahub/core/validators/checks_runner.py and MERGEABLE_CONCLUSIONS in backend/infrahub/proposed_change/tasks.py
5. The web UI: frontend/app/src/entities/diff/ui/checks/ and frontend/app/src/entities/proposed-changes/ui/checks.ts

Parts that need extra attention:

• A high-cost render that runs longer than one hour loses its slot, and another high-cost render can start.
• With the limit on, the artifact generation flow waits until its high-cost renders finish, while it only submits the light ones.

How to test

uv run pytest backend/tests/unit/graphql/cost/test_render_slots.py backend/tests/unit/artifacts/test_high_cost.py backend/tests/unit/config/test_config.py backend/tests/unit/core/validators/test_checks_runner.py
uv run pytest backend/tests/component/graphql/cost/test_render_cost.py backend/tests/component/proposed_change/test_validate_artifacts_generation.py backend/tests/component/proposed_change/test_artifact_definition_generate_members.py
cd frontend/app && npx vitest run src/entities/diff/ui/checks src/entities/proposed-changes/ui/checks.test.ts

Local results: the full unit suite passed (2,856 tests), 116 component tests passed, the 7 new Vitest tests passed, and ruff check, ruff format and invoke docs.validate reported no errors.

Impact and rollout

• Backward compatibility: with both settings off, which is the default, renders run as before. ValidatorConclusion has the new value skipped. API clients receive it only when the skip setting is on.
• Performance: see the results above.
• Config/env changes: INFRAHUB_GRAPHQL_COST_LIMIT_HIGH_COST_RENDERS, INFRAHUB_GRAPHQL_COST_HIGH_COST_RENDERS_AT_ONCE and INFRAHUB_GRAPHQL_COST_SKIP_HIGH_COST_ARTIFACT_CHECKS.

Documentation:

• section "Limit the renders of high-cost queries" in query-cost.mdx
• paragraph on skipped checks in proposed-changes/overview.mdx
• section "High-cost renders" in graphql-query-cost.md
• changelog entry +graphql-query-cost-renders.added.md

Checklist

• Tests added/updated
• Changelog entry added
• External docs updated (if user-facing or ops-facing change)
• Internal .md docs updated (internal knowledge and AI code tools knowledge)
• I have reviewed AI generated content

---

Assisted-by: opsmill-dev-pr 0.2.0

🤖 Generated with Claude Code