feat(backend): add Redis Streams message bus driver
Summary
Infrahub runs its message bus on RabbitMQ, in addition to the Redis server it already needs for the cache and the locks. This pull request adds an experimental Redis Streams driver for the message bus, selected with INFRAHUB_BROKER_DRIVER=redis, so that a deployment can run without RabbitMQ. The driver requires Redis 6.2 or later. It connects to a single Redis server, or, through the new INFRAHUB_BROKER_URL setting, to a Redis server that Redis Sentinel manages.
Not decided yet:
- Whether the driver is ready to merge. The branch still contains temporary changes that run all of CI on the Redis driver, and they must be reverted before merge.
- What to do about the problems listed under "Problems found in review and not fixed yet".
How this came together
The commit hashes below are those on the branch after its rebase onto develop on 2026-11-11. Each rebase changes them.
- The work started as a NATS → Redis conversion written with Claude Opus 4.5 (4309c5e0c0, authored on 2025-12-08), loosely modeled on PrefectHQ/prefect-redis.
- Claude Fable 5 then took over and ran a systematic comparison against prefect-redis to find the gaps the conversion had missed. That surfaced and fixed: a message-loss race on group-less
XREAD $reads (73615a840e), broken event broadcast (a shared consumer group delivered each event to only one worker),refresh.settings.*misrouted to the work queue, silently dropped failure retries, missing crash recovery, unbounded stream growth, and the missing Redis default port (fa586f418e). - A second comparison against ag2ai/faststream's Redis package produced two more hardening commits: a catch-all backoff so consumer tasks survive unexpected errors (4ba34f7d94), and RPC timeout enforcement, because a lost reply left callers awaiting a future forever. The timeout originally covered all three broker adapters. During the rebase onto
develop,develop's own RPC timeout replaced it for RabbitMQ,develophad already removed NATS, and the Redis part is now bc4832e450. - Three rounds of PR review hardening followed, replacing in-memory delayed retries with a durable delayed queue (62fdd543b8), isolating and dead-lettering undeliverable entries (37e8216e89, 24afadd3ce), making reply retention consumption-aware instead of cap-based (b4f02fe89b, 502d900290, a07c529bc2), and deduplicating the handler scaffolding (6a84ec50a7).
- Design decisions (broadcast mechanism, retry semantics, idle-claim threshold, trimming strategy, deferred concurrency) were pressure-tested and confirmed interactively before implementation.
How the Redis driver delivers messages
The code is in redis.py.
- Broadcast events (
refresh.registry.*,refresh.settings.*,refresh.git.*) go to the stream{namespace}:events. Every Infrahub process reads the whole stream withXREADand keeps only the routing keys it subscribes to. The stream is capped at about 10,000 entries. - Work items go to the stream
{namespace}:rpcs, which the task workers share through the consumer groupgit-workers. A worker marks the work items it is handling as in progress every 15 seconds, and every 30 seconds each worker claims withXAUTOCLAIMthe work items nobody has marked for a minute: those of a worker that stopped, and deliveries that never reached their worker. A worker that shuts down makes the work items it was handling claimable at once, so the next claim pass of another worker takes them over. A claimed work item delivered more than 5 times is moved to the list{namespace}:rpcs:dead, which keeps the newest 1,000 entries, instead of being run again. The stream is trimmed up to the oldest entry still pending. Every 30 seconds, a worker also deletes the group's consumers that hold no pending entry and have read nothing for 30 minutes, which belong to workers that stopped without their shutdown cleanup. - RPC replies go to one stream per process,
{namespace}:callback:<worker id>. The process trims the entries it has read and keeps renewing the stream's 30-minute expiry, so the stream of a process that stopped without cleaning up deletes itself. - Each entry is handled in a task of its own, as the RabbitMQ client library does for each delivery. A task worker reads the work queue only while fewer than
INFRAHUB_BROKER_MAXIMUM_CONCURRENT_MESSAGESof its work items are running, so the entries it cannot start yet stay with the group for the other workers. Events and RPC replies have no limit, as on RabbitMQ. - A failed work item is retried after a delay. The retry is stored in the sorted set
{namespace}:delayedbefore the failed entry is acknowledged, so it survives a worker restart. A delayed entry that cannot be delivered is tried again every 60 seconds, and after 5 attempts it is moved to the list{namespace}:delayed:dead, which keeps the newest 1,000 entries. - A failed broadcast event is not retried. The driver drops a delayed message bound for the events stream and logs a warning, because every process would receive the retry and each process that failed it would publish another one. RabbitMQ drops these retries too, because its dead-letter exchange routes only work-queue keys.
rpc()waits for thetimeoutargument orINFRAHUB_BROKER_RPC_TIMEOUTseconds (120 by default), then raisesWorkerTimeoutError, as the RabbitMQ driver does since a12d518b2c.
How the Redis driver connects to Redis
The code is in redis_connection.py and config.py.
INFRAHUB_BROKER_URLaccepts the sameredis://,rediss://,redis+sentinel://andrediss+sentinel://URLs asINFRAHUB_CACHE_URL. With a Sentinel URL, the driver asks the Sentinel nodes for the current master and follows a failover. The settings fail to load, so Infrahub does not start, when the URL is set andINFRAHUB_BROKER_DRIVERis notredis.- Without a URL, the driver connects to
INFRAHUB_BROKER_ADDRESSon port 6379 (orINFRAHUB_BROKER_PORT), database 0, withINFRAHUB_BROKER_TLS_ENABLED,INFRAHUB_BROKER_TLS_INSECUREandINFRAHUB_BROKER_TLS_CA_FILE.INFRAHUB_BROKER_USERNAMEandINFRAHUB_BROKER_PASSWORDdefault to the RabbitMQ default userinfrahub/infrahub, and the driver does not send a username or password that is left at that default. - The driver builds its client with the same code as the cache, so it retries a command up to 10 times after a connection error, and checks a connection that has been idle for 20 seconds before using it.
Behavior during a Redis Sentinel failover
test_redis_sentinel.py starts one Redis master, one replica and one Sentinel, and sends RPCs from an API server bus to a task worker bus.
- The tests check that an RPC is answered through Sentinel, after the master is killed, and after a failover requested with
SENTINEL FAILOVERonce Sentinel has demoted the old master. - In a diagnostic run of the requested failover, Sentinel demoted the old master about 10 seconds after promoting the replica, and until then the driver's connections stayed on the old master. Writes that the old master accepts in that window are lost when it becomes a replica of the new master, because Redis replicates asynchronously (Redis Sentinel documentation, section "Consistency under partitions").
- In the same run, after the demotion, the connections that send write commands,
XREADGROUPincluded, moved to the new master. The connections that only read withXREADstayed on the old master, now a replica, and still received the RPC reply through replication.
Changes made while rebasing onto develop
The branch was rebased from stable onto develop on 2026-11-04, and the pull request now targets develop. On 2026-11-11, it was rebased again onto develop at 33100c4c1b.
developremoved the NATS message bus driver, so this branch no longer changes it.developadded its own RPC timeout in a12d518b2c. The RabbitMQ and local drivers keep that code, and the Redis driver'srpc()follows the same rules (bc4832e450). The branch's RabbitMQ RPC timeout integration test was dropped, becausetest_rpc_timeout.pyondevelopcovers the same behavior.developupgradedredisto 8.1.0 and enables the ruff rule against catchingExceptionwithout a reason. f38a911e30 adapts the driver to both, and fixes the TLS settings, which the client ignored before: it never loadedINFRAHUB_BROKER_TLS_CA_FILEand still verified the certificate whenINFRAHUB_BROKER_TLS_INSECUREwas set.developaddedINFRAHUB_CACHE_MAX_CONNECTIONSin f29e3a59b7, which sets the size of the cache and lock connection pools. 415e7783de keeps that size for the cache client, and the configuration still warns when amax_connectionsparameter inINFRAHUB_CACHE_URLoverrides it. The message bus client is built without the setting, so its pool keeps the redis-py default of 100 connections.
Changes outside the driver
redis_connection.py: the cache and the broker share the code that builds a single-node URL, andbuild_broker_connection_string()andbuild_broker_redis_connection()are new (415e7783de).config.py:BrokerSettings.urlis new, the username and password settings have descriptions, and the globaltls.ca_bundlefillsbroker.tls_ca_filewhen a broker URL is set, as it does for the cache URL.- Generated files: the configuration reference and the environment block of the root
docker-compose.yml, which now listsINFRAHUB_BROKER_URL. message_bus/__init__.py: the RPC publish-and-wait logic and the request id enricher, which the RabbitMQ and Redis drivers both used to carry, are written once here. The newlisten(topic, listener)method passes the messages matching an AMQP topic to a listener, whichinfrahub events listennow uses with either driver (4949485e0d).- The knowledge doc
message-bus.mdand the changelog fragment+redis-message-bus.added.md.
Temporary changes that run CI on the Redis driver (revert before merge)
ci.ymlsetsINFRAHUB_BROKER_DRIVER=redisfor every job.E2E-testing-invoke-demo-startstays on RabbitMQ, because it starts a released Infrahub image, which has no Redis driver.version-upgrade.ymlstays on RabbitMQ, because it starts Infrahub 1.8.0 before upgrading.development/docker-compose.ymldefaults the driver toredisand the broker address tocache.- The
infrahub-testcontainerscompose file, its cluster variant andcontainer.pydefault the driver toredisand the broker address tocache(05a2032eb7, 92bfbf95cb):- The RabbitMQ service
message-queuebelongs to the compose profilerabbitmq, so CI stacks no longer start it. - With
INFRAHUB_BROKER_DRIVER=rabbitmq, the env file turns on that profile and points the broker address atmessage-queue. infrahub-serverdepends onmessage-queuewithrequired: false, which needs Docker Compose 2.20 or later.
- The RabbitMQ service
- The RabbitMQ integration tests skip when the driver is not
rabbitmq.
Problems found in review and fixed
- 2901215fa1: a failed broadcast event was retried onto the events stream, so the number of runs grew with every retry round; a stream maintenance command that the server kept refusing stopped the consumer from reading its stream; and the default RabbitMQ credentials were sent to Redis, which refused a Redis server without that user.
- 92bfbf95cb: the testcontainers stacks could start RabbitMQ but could not log in to it, because the compose files set empty broker credentials.
- 4949485e0d:
- Each stream consumer ran one handler at a time, so a long git flow delayed the events and work items behind it, and
INFRAHUB_BROKER_MAXIMUM_CONCURRENT_MESSAGEShad no effect. - The consumers of workers that stopped without a clean shutdown stayed in the
git-workersgroup. - On Redis 6.2,
EXPIRE ... NXfailed after each RPC reply was written, and the required Redis version was not documented. infrahub events listenonly read from RabbitMQ.- Two comments in
redis.pydescribed behavior the code does not have, several policy values were literals, and the RPC wait and header building were written twice.
- Each stream consumer ran one handler at a time, so a long git flow delayed the events and work items behind it, and
- 549e9eb7b7:
- A work item whose worker stopped before acknowledging it, or whose
XREADGROUPreply was lost, was delivered again only after 30 minutes. It is now taken over once nobody has marked it as in progress for a minute, and within 30 seconds when its worker shut down. - A work item whose handler kept stopping its worker was passed from worker to worker without a limit. It now goes to the dead-letter list after 5 deliveries. The RabbitMQ driver has no delivery limit with its default classic queues, which do not support poison message handling (RabbitMQ documentation, "Classic Queue Features"); quorum queues drop or dead-letter a message after 20 deliveries by default since RabbitMQ 4.0 (RabbitMQ documentation, "Poison Message Handling").
- A work item whose worker stopped before acknowledging it, or whose
Problems found in review and not fixed yet
A code review of this pull request on 2026-11-05 reported the following. They are not fixed, and they have not been reproduced.
- When the consumer group is missing and re-creating it keeps failing, the consumer retries without waiting. Re-creating the group at id
0delivers again every entry still in the stream. This is left as it is for now. - The RabbitMQ integration tests also skip when
INFRAHUB_USE_TEST_CONTAINERS=false, so they no longer run against a RabbitMQ that is already running.
How this was tested
- Unit tests:
test_topic_matches.py,test_redis_connection.py,test_config.py,test_tls_settings.pyand the Redis bus routing test. - Integration tests: 37 in
test_redis.pyagainst one Redis server, and 3 intest_redis_sentinel.pyagainst a master, a replica and a Sentinel started with host networking. The twolistencases intest_rabbitmq.pyrun only withINFRAHUB_BROKER_DRIVER=rabbitmq. - Redis 6.2, run locally: the Sentinel tests pass, and the
test_redis.pytests pass when run one at a time, including the claim, hand-over and delivery limit tests, excepttest_redis_consumer_deregistered_on_shutdown, which expectsXREADGROUPto create a consumer on an empty read; Redis 6.2 and 7.0 do not do that. - CI passed on the latest commit, 1f3b0d169a (run 43218811049).
uv run pytest backend/tests/unit/test_redis_connection.py backend/tests/unit/config backend/tests/unit/services/adapters/message_bus
uv run pytest backend/tests/integration/services/adapters/message_bus/test_redis.py backend/tests/integration/services/adapters/message_bus/test_redis_sentinel.py
create-dataset runs: same duration as develop, and Redis storage stays bounded
On 2026-11-11, the TestCreateDataset test of infrahub-private-tests ran locally on one machine (24 CPUs, 80 GB of memory) with the small profile. Each run used an enterprise image built from infrahub-private develop, the same way the create-dataset workflow builds it. The results were not uploaded.
- Duration, from two runs of
develop(33100c4c1b) with RabbitMQ and two of this branch (cd5465b536) with Redis:- In the pair where the stack stayed up until the computed-attribute flows stopped, the last of those flows started 1918 s (RabbitMQ) and 1917 s (Redis) after
test_load_initial_datastarted. - Both runs of that pair started 5125
computed_attribute_process_transformflows and 2726query-computed-attribute-transform-targetsflows. - The data load alone varies between runs of the same image: 1155 s to 1676 s with RabbitMQ, and 1235 s to 1509 s over three runs with Redis.
- In the pair where the stack stayed up until the computed-attribute flows stopped, the last of those flows started 1918 s (RabbitMQ) and 1917 s (Redis) after
- Redis storage, from a third run of this branch that recorded the Redis keys every 30 seconds for 90 minutes:
- The run covered the data load, 10 idle minutes, and a restart of every task worker and API server (one of each with SIGTERM, the other with SIGKILL).
- The message bus keys never used more than 54 KB.
- Only
{namespace}:eventskept growing, by about one 584-byte entry per minute. The stream is capped at about 10,000 entries (redis.py). - The two consumers that the restarted task workers left in the
git-workersgroup were removed 30 minutes after the restart.
Found in these runs and not fixed:
- A task worker does not shut the message bus down when it stops, because only the API server calls the services' shutdown (
server.py). The consumer of a stopped task worker, and its callback stream if it has one, stay until the 30-minute cleanup. - While the handler of one work item never finishes, its worker keeps marking the item as in progress.
{namespace}:rpcsthen keeps every later entry, because it is trimmed only up to the oldest pending entry (_trim_worked_stream). This was not reproduced. - Two of the three Redis runs logged
Redis error in consumerwithTimeoutError: Timeout reading from cache:6379a few seconds after a process started loading schemas. redis-py 8.1 uses a 5-second read timeout by default. The cause is not known.
Traces of the refresh.registry.branches broadcast: one producer span, one consumer span in each process
Since fbbccb7b2e, the driver opens a producer span when it publishes a message (publish_message) or an RPC reply (publish_reply), and a consumer span (on_callback, on_message) in each process that handles an entry, under the trace context read from the entry headers (_producer_span, _traced_message). Besides routing_key, the spans carry these OpenTelemetry messaging attributes:
messaging.system(redis),messaging.destination.name(the stream),messaging.message.id(the stream entry ID) andmessaging.operation.type(sendorprocess)messaging.consumer.group.nameon a work itemmessaging.message.conversation_idon an RPC request and its reply
Before fbbccb7b2e, the spans had the kind INTERNAL and only the routing_key attribute, and sending a reply opened no span.
On 2026-11-12, a local development stack of fbbccb7b2e ran with the Redis driver, the Tempo and Grafana services of docker-compose-observability.yml, and INFRAHUB_TRACE_SAMPLE_RATIO=1.0. It ran one API server with 4 worker processes and 2 task workers. Alloy and cAdvisor were left out, because Docker could not mount /var/run/docker.sock into a container on that machine; they collect logs and metrics, not traces.
- Schema update, with
infrahubctl schema loadof a schema with one node kind: in the trace ofPOST /api/schema/load, the API server process that handled the request has apublish_messageproducer span forrefresh.registry.branches. The span has 6on_callbackconsumer children, one in each process of the stack, the publishing process included. - Branch merge, with
infrahubctl branch mergeof a branch with one new node: in the trace of the GraphQLPOST, the merge flow (Merge branch trace-merge-redis-kinds into main) on a task worker has thepublish_messageproducer span. The span again has 6on_callbackconsumer children, one in each process. - Tempo's service graph recorded
messaging_systemedges frominfrahub-task-workertoinfrahub-task-workerfor these spans. With the INTERNAL spans of the earlier runs, it recorded none. - An RPC request and its reply are not part of these two operations.
test_redis_rpc_is_traced_through_the_request_and_the_replychecks them against a Redis server:publish_messagein the caller,on_messagein the task worker as its child,publish_replyunderon_message, andon_callbackin the caller as the child ofpublish_reply.
In each pair of images below, the first one lists the result of a TraceQL query: the publish_message span whose parent is the request or the merge flow, and the on_callback spans whose parent is publish_message, with the kind, service, process id (worker.id), stream and entry ID of each span. The second one is the trace view filtered to the same spans, with the details of publish_message open. Grafana prints the service name on some rows of the trace view only; the first image lists the service of every span.
Schema update:


Branch merge:

