fix(graphql): retry transient database errors on object create

Closed
BE
1mo ago
+99-8

Fixes #10119

Problem

mutate_create was the only CRUD mutation on InfrahubMutationMixin not wrapped in @retry_db_transaction:

MutationDecorator before this PR
mutate_createnone
mutate_update@retry_db_transaction(name="object_update")
mutate_upsert@retry_db_transaction(name="object_upsert")
mutate_delete@retry_db_transaction(name="object_delete")

The IPAM mutations already retry their creates (ipaddress_create, ipprefix_create in graphql/mutations/ipam.py), which left the generic create path as the sole outlier.

The practical effect is that a transient Neo4j error during an object create is handed to the user verbatim:

there are no available threads to serve this request at the moment, you can retry at a
later time or consider increasing max thread pool size for bolt connectors

Under identical load, an update, upsert or delete recovers transparently. This came out of a support engagement on a production-scale deployment where the aggregate driver connection pools across server and task-worker processes outran Neo4j's configured bolt thread count. Increasing the bolt thread pool is a reasonable deployment-side mitigation, but the inconsistency is a defect on its own: transient contention should not be user-visible on exactly one CRUD verb.

Change

One line in backend/infrahub/graphql/mutations/main.py, applied exactly as on mutate_update (below @classmethod):

@classmethod
@retry_db_transaction(name="object_create")
async def mutate_create(

Why this is retry-safe

  • create_node (core/node/create.py) wraps the node writes in db.start_transaction(), so a TransientError rolls the attempt back and the retry starts from a clean slate.
  • File uploads are not repeated: _process_file runs in mutate() before mutate_create is called, so it sits outside the retried scope.
  • Mutation events are emitted after mutate() returns, also outside the retried scope.
  • The create body already runs under retry today on another path — mutate_upsert is decorated and calls cls.mutate_create directly. This PR makes the two entry points behave the same instead of differing by how the create was reached.

Two notes for reviewers, both pre-existing rather than introduced here:

  • Nested retry on the upsert path. mutate_upsert (retried) calling mutate_create (now retried) means the create leg can attempt up to retry_limit² times. This already happens for IPAM kinds, whose generated Upsert inherits the retried mutate_upsert and dispatches to an already-retried mutate_create. Worth flagging if you'd rather bound it, but it is not new behaviour.
  • Post-commit profile application. create_node applies profiles and re-saves after the transaction commits. A transient error in that narrow window would be retried and could duplicate the node. That window is identical on the already-retried upsert and IPAM-create paths, so this PR neither introduces nor widens it.

Tests

New backend/tests/unit/graphql/test_mutation_retry.py:

  • test_mutate_create_retries_transient_error — drives mutate_create with create_node raising TransientError once, asserts the call succeeds on the second attempt and that TRANSACTION_RETRIES{name="object_create"} incremented. This also pins the metric label.
  • test_crud_mutations_are_retry_wrapped — parametrized over all four CRUD mutations, so none can silently lose its retry again.

Verified red/green: both new tests fail with the decorator removed and pass with it. Full run of backend/tests/unit/database/test_retry_db_transaction.py plus the new file is 14 passed. ruff check and ruff format --check are clean on both touched files.

Release line

Targeted at stable so the fix ships on the released line and merges forward into release-1.11 with the usual sync.

🤖 Generated with Claude Code