feat(database): make the Neo4j connection pool size configurable
Why
Infrahub built the Neo4j driver with no pool sizing arguments, so the neo4j driver's own default of 100 connections per remote address applied, and DatabaseSettings exposed no setting to change it. Because every process builds its own driver, the ceiling is per process and multiplies with the deployment size, while Neo4j's bolt thread pool does not.
On a production-scale deployment seen during a support engagement — 3 server pods at 4 Gunicorn workers each, plus 3 task workers — that is 15 processes x 100 = up to 1,500 potential bolt connections against Neo4j's default server.bolt.thread_pool_max_size of 400, roughly 3.75x oversubscription. Under load Neo4j stops queuing and starts rejecting: There are no available threads to serve this request at the moment. The only mitigations available were on the Neo4j side or by cutting the process count; there was no Infrahub-side knob at all.
Goal: give operators a supported way to bound the connection footprint, and pick a default that is safe for multi-process deployments out of the box.
Non-goals: this does not add admission control, request shedding, or any other backpressure mechanism. It does not touch connection_acquisition_timeout.
Closes #10120
What changed
Behavioral changes:
- New setting
INFRAHUB_DB_MAX_CONNECTION_POOL_SIZE(DatabaseSettings.max_connection_pool_size, minimum 1), passed through toAsyncGraphDatabase.driver()inget_db(). - The default is 100 — identical to the driver's previous implicit default, so the change is purely additive with zero behavior change. It is also the value
release-1.11already ships for this setting (added by #9879).
Implementation notes:
- The setting and its wiring are extracted from the unmerged branch
dga/feat-rate-limiting-api-zb38x, where they were introduced by @dgarros as part of a larger API backpressure and admission-control effort. Only the pool setting is taken here; the admission-control, concurrency-factor, and stress-shedding settings from that branch are deliberately left out. Commit8578575on that branch chose 50, but the merged form of that work onrelease-1.11(#9879) ships 100 — this PR matches release-1.11, and theconfig.pyfield block is byte-identical to it so the periodic stable-into-release merge resolves cleanly. docs/docs/reference/configuration.mdxis generated from the settings classes, so it was regenerated withinvoke docs.generate-configrather than hand-edited. The regenerated diff is exactly one row.
What stayed the same: no schema changes, no API contract changes, no other driver arguments touched.
How to review
The default-value question raised in the first revision is settled by prior art: release-1.11 already ships this exact setting with default=100, added by #9879 (the merged form of the rate-limiting branch). This PR now matches it — the config.py field block is byte-identical to release-1.11's, so the periodic stable→release-1.11 merge resolves cleanly, and there is no behavior change on stable (100 was already the driver's implicit default). Operators bound the footprint by lowering the value; sizing guidance is under Impact & rollout.
Files: backend/infrahub/config.py and backend/infrahub/database/__init__.py are the whole fix. docs/docs/reference/configuration.mdx is generated. The tests are the rest.
How to test
# unit tests for the setting and the driver wiring
uv run pytest backend/tests/unit/config/test_config.py backend/tests/unit/database/ -q
# the docs table is generated -- confirm it is in sync
uv run invoke docs.generate-config && git diff --exit-code docs/docs/reference/configuration.mdx
# lint
uv run ruff check backend/infrahub/config.py backend/infrahub/database/__init__.py
backend/tests/unit/database/test_get_db.py is the regression guard for the wiring: it patches the driver factory and asserts the kwarg is forwarded. Verified that it fails with KeyError: 'max_connection_pool_size' when the get_db() line is removed, and passes with it.
Locally: 69 passed (58 after fragment removal re-run), ruff clean, vale clean on the changelog fragment.
Impact & rollout
- Backward compatibility: none — the explicit default (100) equals the driver's previous implicit default. Sizing guidance: total connections is
(server pods x Gunicorn workers + task workers) x this value, and that product should stay under Neo4j'sserver.bolt.thread_pool_max_size. - Performance: unchanged at the default. Deployments that lower the value trade connection-acquisition waiting for bounded pressure on Neo4j — the intended knob for the deployment shape that triggered the incident.
- Config/env changes: one new variable,
INFRAHUB_DB_MAX_CONNECTION_POOL_SIZE, documented in the generated config reference. - Deployment notes: safe to deploy; takes effect on process restart since the driver is built at startup.
- Release line: this targets
stable(the 1.10.x line, currently 1.10.6), which is where the incident was observed, and rides forward torelease-1.11through the usual stable-into-release merges. The repository has no backport label or workflow, so there is nothing to apply — flagging the intent here instead.
Checklist
- Tests added/updated
- Changelog entry added (
changelog/10120.added.md,changelog/10120.changed.md) - External docs updated (regenerated, not hand-edited)
- Internal .md docs updated (no internal docs cover driver pool sizing)
- I have reviewed AI generated content
🤖 Generated with Claude Code