fix(graphql): retry transient database errors on object create
Fixes #10119
Problem
mutate_create was the only CRUD mutation on InfrahubMutationMixin not wrapped in @retry_db_transaction:
| Mutation | Decorator before this PR |
|---|---|
mutate_create | none |
mutate_update | @retry_db_transaction(name="object_update") |
mutate_upsert | @retry_db_transaction(name="object_upsert") |
mutate_delete | @retry_db_transaction(name="object_delete") |
The IPAM mutations already retry their creates (ipaddress_create, ipprefix_create in graphql/mutations/ipam.py), which left the generic create path as the sole outlier.
The practical effect is that a transient Neo4j error during an object create is handed to the user verbatim:
there are no available threads to serve this request at the moment, you can retry at a
later time or consider increasing max thread pool size for bolt connectors
Under identical load, an update, upsert or delete recovers transparently. This came out of a support engagement on a production-scale deployment where the aggregate driver connection pools across server and task-worker processes outran Neo4j's configured bolt thread count. Increasing the bolt thread pool is a reasonable deployment-side mitigation, but the inconsistency is a defect on its own: transient contention should not be user-visible on exactly one CRUD verb.
Change
One line in backend/infrahub/graphql/mutations/main.py, applied exactly as on mutate_update (below @classmethod):
@classmethod
@retry_db_transaction(name="object_create")
async def mutate_create(
Why this is retry-safe
create_node(core/node/create.py) wraps the node writes indb.start_transaction(), so aTransientErrorrolls the attempt back and the retry starts from a clean slate.- File uploads are not repeated:
_process_fileruns inmutate()beforemutate_createis called, so it sits outside the retried scope. - Mutation events are emitted after
mutate()returns, also outside the retried scope. - The create body already runs under retry today on another path —
mutate_upsertis decorated and callscls.mutate_createdirectly. This PR makes the two entry points behave the same instead of differing by how the create was reached.
Two notes for reviewers, both pre-existing rather than introduced here:
- Nested retry on the upsert path.
mutate_upsert(retried) callingmutate_create(now retried) means the create leg can attempt up toretry_limit²times. This already happens for IPAM kinds, whose generatedUpsertinherits the retriedmutate_upsertand dispatches to an already-retriedmutate_create. Worth flagging if you'd rather bound it, but it is not new behaviour. - Post-commit profile application.
create_nodeapplies profiles and re-saves after the transaction commits. A transient error in that narrow window would be retried and could duplicate the node. That window is identical on the already-retried upsert and IPAM-create paths, so this PR neither introduces nor widens it.
Tests
New backend/tests/unit/graphql/test_mutation_retry.py:
test_mutate_create_retries_transient_error— drivesmutate_createwithcreate_noderaisingTransientErroronce, asserts the call succeeds on the second attempt and thatTRANSACTION_RETRIES{name="object_create"}incremented. This also pins the metric label.test_crud_mutations_are_retry_wrapped— parametrized over all four CRUD mutations, so none can silently lose its retry again.
Verified red/green: both new tests fail with the decorator removed and pass with it. Full run of backend/tests/unit/database/test_retry_db_transaction.py plus the new file is 14 passed. ruff check and ruff format --check are clean on both touched files.
Release line
Targeted at stable so the fix ships on the released line and merges forward into release-1.11 with the usual sync.
🤖 Generated with Claude Code