feat(backend): store the cost level of stored GraphQL queries [INFP-739]
Stacked pull requests for INFP-739. Each one is based on the one before it:
- #11004 (this pull request): Store the cost level of each stored GraphQL query. Base: #11003 (
fac/graphql-query-cost-impl-rrake), which adds the query cost estimate. - #11005: Warn about a high-cost stored query in the web UI.
- #11006: Limit or skip the artifact renders of high-cost queries.
Why: a stored GraphQL query has no cost to read
A team that adds a stored GraphQL query to an artifact definition cannot view, before Infrahub renders the artifacts, whether one run of the query reads a large part of the database. The base branch estimates the cost of a request when the client sends the X-Infrahub-Query-Cost: details header or runs InfrahubGraphQLQueryReport, but Infrahub does not keep an estimate on the stored query. The two next pull requests read one: the warning in the web UI and the limit on artifact renders.
Not in this pull request:
- the warning in the web UI (next pull request)
- any change to how artifacts render (last pull request)
Cost level and cost estimate of a stored query
CoreGraphQLQueryhas two new read-only, optional attributes:cost_level:highwhen the worst-case database rows estimated for one run are aboveINFRAHUB_GRAPHQL_COST_HIGH_COST_DATABASE_ROWS(new setting, 250,000 by default),lowwhen they are not, andunknownwhen the query cannot be estimated, for example when it contains a mutation or when no statistics exist yet.cost_estimate: the figures behind the level, which are the expected and worst-case nodes, resolver calls and database rows, the limit, when the statistics were computed, the three fields with the most worst-case database rows, and how many fields have no estimate.
- Infrahub estimates a stored query when a user creates or updates it, and again after each refresh of the statistics (the daily
graphql-cost-statistics-refreshflow). After a refresh, it saves only the stored queries of the default branch whose level or figures changed.
Python SDK protocols with the two attributes
The python_sdk submodule points at opsmill/infrahub-sdk-python#1414, which adds cost_level and cost_estimate to the CoreGraphQLQuery protocols. Without it, invoke backend.validate-generated finds the SDK protocols out of date. That commit is on top of the SDK stable branch, so the submodule update also brings in these SDK commits:
- six dependency updates, which change only the lock file of the SDK
- opsmill/infrahub-sdk-python#1293, which makes generators, transforms and checks use the configured default branch
Why the level uses the worst-case database rows
A stored query has no variable values, so the estimate covers every node of the queried kinds and the expected figure is an average over all of them. Measured on a test data set with one logical network of 938 devices and ten of 80 devices:
- One run of a cabling plan query took 80 seconds for the large network and 4.4 seconds for a small one.
- Estimated from the statistics only, the query has 99,188 expected and 1,138,406 worst-case database rows. Only the worst case is above the 250,000 default.
- Estimated with the variables of each target, the worst case is 991,262 database rows for the large network and 112,806 for a small one. The default lies between the two. The last pull request of the stack uses this estimate for each target.
- The estimated database rows were 3.5 times the actual rows for the large network and 6 times for a small one. The limit is in estimated rows.
How to review
backend/infrahub/graphql/cost/stored_query.py: the estimate and the levelbackend/infrahub/graphql/mutations/graphql_query.py: the estimate when a query is savedbackend/infrahub/graphql/cost/tasks.py: the estimate after a refresh of the statistics
Generated files, no review needed: schema/schema.graphql, backend/infrahub/core/protocols.py, backend/tests/protocols.py, the frontend gql.tada types, docs/docs/reference/configuration.mdx and the two docker-compose.yml files.
How to test
uv run pytest backend/tests/unit/graphql/cost/test_stored_query.py
uv run pytest backend/tests/component/graphql/cost/test_stored_query_cost.py
Local results: 214 tests passed in the query cost suites and the suites they touch, and the full unit suite passed (2,845 tests). With the submodule update, invoke backend.validate-generated passed and the full unit suite passed again.
Impact and rollout
- Backward compatibility: no graph migration. The core schema update adds the two optional attributes. Existing stored queries have no level until a user saves them or the statistics are refreshed. Not tested: the upgrade of an existing database.
- Performance: one estimate when a user saves a stored query, and one estimate for each stored query of the default branch after each refresh of the statistics. Not measured.
- Config/env changes:
INFRAHUB_GRAPHQL_COST_HIGH_COST_DATABASE_ROWS, 250,000 by default. - Deployment notes: merge opsmill/infrahub-sdk-python#1414 first. If it is squash-merged, move the
python_sdksubmodule to the merged commit before this pull request merges.
Documentation:
- section "View the cost level of a stored query" in query-cost.mdx
- section "Stored query cost" in graphql-query-cost.md
- changelog entry +graphql-query-cost-level.added.md
Checklist
- Tests added/updated
- Changelog entry added
- External docs updated (if user-facing or ops-facing change)
- Internal .md docs updated (internal knowledge and AI code tools knowledge)
- I have reviewed AI generated content
Assisted-by: opsmill-dev-pr 0.2.0
š¤ Generated with Claude Code