Pull requests for INFP-739:

1. #11004: Store the cost level of each stored GraphQL query. Base: #11003 (fac/graphql-query-cost-impl-rrake), which adds the query cost estimate.
2. #11005: Warn about a high-cost stored query in the web UI.
3. Two alternatives, both based on #11005:
   • #11006: Limit or skip the artifact renders of high-cost queries.
   • This pull request: Run the database queries of high-cost requests with the pipelined runtime of Neo4j.

Why: the database queries of high-cost renders can use several CPU cores each

When a proposed change or a merge affects the artifacts of a high-cost stored query, Infrahub submits all their renders at once, and each render runs the query. Infrahub Enterprise runs every query that reads data outside a transaction with the parallel runtime of Neo4j, which uses several CPU cores of the database server for one query. This includes the queries that a GraphQL query sends for each of its fields. #11006 runs fewer high-cost renders at once, or leaves them out of the checks of a proposed change. This pull request keeps the renders running at the same time, and runs each database query of a high-cost render on one core.

Not in this pull request:

• Requests to /graphql: Infrahub does not estimate them, so that the estimate queries do not run on every request.
• Any change in Infrahub Community: Community runs no query with the parallel runtime, so the setting has no effect there.

One setting, off by default

With INFRAHUB_GRAPHQL_COST_PIPELINED_RUNTIME_FOR_HIGH_COST_QUERIES=true on the API servers, Infrahub handles each request to a stored query with the high level at /api/query/<name> as follows:

1. It estimates the request with its variables. It reuses the estimate when the client asked for the cost details.
2. When the worst-case database rows are above INFRAHUB_GRAPHQL_COST_HIGH_COST_DATABASE_ROWS, or when the request cannot be estimated, the database queries of the GraphQL execution that would use the parallel runtime use the pipelined runtime. The pipelined runtime runs each query on one core.
3. It logs Running the stored GraphQL query <name> with the pipelined Cypher runtime with the estimated worst-case database rows. The runtime label of the infrahub_db_query_execution_seconds metric shows pipelined for these queries.

Requests to stored queries with the low or unknown level, and requests with a mutation, are not estimated and keep their runtime.

How the runtime is switched

• The endpoint sets a ContextVar around the GraphQL execution, so the resolvers and the tasks they start read the same value.
• InfrahubDatabase replaces only Neo4jRuntime.PARALLEL, whether it comes from the configuration of the query or from the default runtime, which Infrahub Enterprise sets to parallel. Write queries do not change, because Infrahub sends them without a runtime prefix.

Results on a test data set

On Infrahub 1.11.5 Enterprise with this stack, on a host with 32 cores, with one logical network of 938 devices and ten of 80 devices. Three artifact definitions render the stored query for every network. Runs 1 and 2 without settings, the limit runs and pipelined run 1 ran on one day; runs 3 and 4 without settings and pipelined runs 2 and 3 ran on the next day. The probe is a light GraphQL query sent every 5 seconds; its column shows how long the API took to answer it. The test reads the CPU time of each container from Docker every 5 seconds. CPU usage is in percent of one core (100 % is one core busy for the whole period): the mean over the phase and the peak over 5 seconds. "Whole stack" includes the cache, the message queue and the other services of the test stack.

With the default limit of 250,000 worst-case database rows, only the 3 renders for the large network ran with the pipelined runtime: their estimate is 991,262 rows, and the estimate for each small network is 112,806 rows.

Checks of the proposed change:

Settings	Duration	Probe, 95th percentile / longest	Database, mean / peak	API servers, mean / peak	Task workers, mean / peak	Whole stack, mean / peak
None, run 1	182.7 s *	1.3 s / 11.4 s	71 % / 458 %	254 % / 758 %	12 % / 50 %	370 % / 1,012 %
None, run 2	113.4 s	3.2 s / 4.2 s	103 % / 383 %	339 % / 801 %	14 % / 52 %	496 % / 969 %
None, run 3	118.1 s	5.0 s / 10.7 s	102 % / 410 %	358 % / 801 %	15 % / 59 %	517 % / 1,035 %
None, run 4	116.0 s	1.5 s / 5.4 s	104 % / 426 %	338 % / 756 %	14 % / 56 %	496 % / 957 %
Limit (#11006)	243.5 s	0.01 s / 4.9 s	49 % / 423 %	163 % / 738 %	8 % / 56 %	249 % / 949 %
Limit and skip (#11006)	41.3 s	6.0 s / 6.0 s	187 % / 401 %	333 % / 749 %	24 % / 49 %	603 % / 931 %
Pipelined (this pull request), run 1	125.8 s	3.6 s / 7.2 s	92 % / 355 %	314 % / 791 %	13 % / 58 %	456 % / 1,054 %
Pipelined (this pull request), run 2	183.2 s *	2.2 s / 5.0 s	70 % / 374 %	266 % / 801 %	11 % / 46 %	381 % / 1,057 %
Pipelined (this pull request), run 3	112.3 s	1.5 s / 1.5 s	101 % / 383 %	348 % / 802 %	14 % / 48 %	504 % / 1,104 %

Merge until all artifacts are generated:

Settings	Duration	Probe, 95th percentile / longest	Database, mean / peak	API servers, mean / peak	Task workers, mean / peak	Whole stack, mean / peak
None, run 1	111.5 s	1.8 s / 5.3 s	57 % / 247 %	379 % / 754 %	11 % / 55 %	483 % / 975 %
None, run 2	112.8 s	0.8 s / 2.3 s	59 % / 251 %	374 % / 799 %	13 % / 51 %	482 % / 1,003 %
None, run 3	118.3 s	0.6 s / 4.2 s	59 % / 320 %	353 % / 779 %	12 % / 54 %	459 % / 995 %
None, run 4	172.3 s *	0.15 s / 3.8 s	39 % / 204 %	242 % / 788 %	9 % / 45 %	320 % / 1,035 %
Limit (#11006)	237.1 s	0.01 s / 0.2 s	27 % / 273 %	164 % / 700 %	7 % / 50 %	225 % / 969 %
Limit and skip (#11006)	227.3 s	0.01 s / 1.7 s	28 % / 209 %	168 % / 787 %	7 % / 68 %	233 % / 994 %
Pipelined (this pull request), run 1	178.9 s *	0.10 s / 10.0 s	40 % / 200 %	232 % / 793 %	9 % / 66 %	311 % / 1,042 %
Pipelined (this pull request), run 2	106.2 s	2.2 s / 5.0 s	59 % / 195 %	362 % / 725 %	12 % / 47 %	468 % / 1,050 %
Pipelined (this pull request), run 3	118.0 s	6.9 s / 10.0 s	63 % / 261 %	349 % / 794 %	11 % / 44 %	457 % / 1,025 %

* This phase took 55 to 75 s longer than the other phases with the same settings. Run 4 without settings recorded the time of each render: two of the three renders for the large network ran 57 and 74 s longer than the third. The earlier runs did not record these times.

Why some phases take 55 to 75 s longer

• Each render for the large network keeps one API worker process at about 100 % of a CPU core for 80 to 95 s. Each of the two API servers of the test stack runs 4 worker processes, and a worker process runs its Python code on one core at a time.
• When two of the three renders for the large network go to the same worker process, they share its core, and both take about 1.7 times as long. If each render went to one of the 8 processes at random, this would happen in 34 % of the phases. On the second day, 7 of the 20 phases in which every render ran at once took 55 to 75 s longer, counting runs with other settings that are not in this pull request.
• In two slow phases, the CPU time of each API worker process, read every 5 seconds, showed one process at 80 % of a core or more for 148 s and 153 s, and another for 82 s and 87 s. In a fast phase, three processes were each busy for 87 to 92 s.
• So the duration of one run does not show the effect of a setting. The other phases took 106 to 126 s, with and without the pipelined runtime.

What the pipelined runtime changes

• The API servers used the most CPU in every run: up to 8 cores, against at most 4.6 cores for the database. The pipelined runtime does not change the work of the API servers, so the peak of the whole stack stayed at about 10 cores.

• The pipelined runtime did not lower the total CPU time of the whole stack beyond the variation between runs (mean usage multiplied by duration):

  • checks of the proposed change: 574, 698 and 566 CPU seconds, against 676, 562, 611 and 575 without settings and 606 with the limit
  • merge: 556, 497 and 539 CPU seconds, against 539, 544, 543 and 551 without settings, 533 with the limit and 530 with limit and skip

  A slow phase spreads about the same CPU time over a longer time, so its mean usage is lower. With limit and skip, the checks of the proposed change used 249 CPU seconds, because they did not render the artifacts of the large network.

• In the checks of the proposed change, the peak of the database was 355 to 383 % with the pipelined runtime, against 383 to 458 % without settings. After the merge, the ranges overlap: 195 to 261 % against 204 to 320 %.

• The probe answers vary too much between runs with the same settings to compare the settings: without settings, the longest answer during the checks of the proposed change was between 4.2 s and 11.4 s.

• With limit and skip, the probe answered only 8 times during the checks of the proposed change.

How to review

1. backend/infrahub/graphql/cost/high_cost_request.py: which requests are high-cost
2. backend/infrahub/graphql/cost/runtime.py and the change in backend/infrahub/database/__init__.py: how the runtime is replaced
3. backend/infrahub/api/query.py: where the endpoint estimates the request and switches the runtime

Parts that need extra attention:

• When the setting is on, Infrahub estimates each request to a stored query with the high level before it runs the query, including the requests that end up below the limit.
• In a transaction, Infrahub sends a read query with the parallel runtime without a runtime prefix. During a high-cost request, such a query gets the pipelined prefix instead.
• The ContextVar also applies to tasks that the GraphQL execution starts. A task that outlives the request keeps the pipelined runtime.

How to test

uv run pytest backend/tests/unit/graphql/cost/test_high_cost_request.py backend/tests/unit/database/test_parallel_runtime_replacement.py
uv run pytest backend/tests/component/api/test_query_pipelined_runtime.py

Local results: the 11 new unit tests and the 5 new component tests passed, the full unit suite passed (2,856 tests), the 20 component tests of the stored query endpoint passed, and ruff check, ruff format and invoke docs.validate reported no errors. In the performance test, the metrics of the API servers showed node, attribute, relationship and peer queries with runtime="pipelined".

Impact and rollout

• Backward compatibility: with the setting off, which is the default, every query keeps its runtime.
• Performance: see the results above.
• Config/env changes: INFRAHUB_GRAPHQL_COST_PIPELINED_RUNTIME_FOR_HIGH_COST_QUERIES, added to both compose files with the default value.

Documentation:

• section "Run the database queries of high-cost requests on one CPU core" in query-cost.mdx
• section "Pipelined runtime for high-cost requests" in graphql-query-cost.md
• setting in configuration.mdx
• changelog entry +graphql-query-cost-pipelined.added.md

Checklist

• Tests added/updated
• Changelog entry added
• External docs updated (if user-facing or ops-facing change)
• Internal .md docs updated (internal knowledge and AI code tools knowledge)
• I have reviewed AI generated content

---

Assisted-by: opsmill-dev-pr 0.2.0

🤖 Generated with Claude Code