feat(backend): add Redis Streams message bus driver

FA
8d ago
+3533-110

Summary

Infrahub runs its message bus on RabbitMQ, in addition to the Redis server it already needs for the cache and the locks. This pull request adds an experimental Redis Streams driver for the message bus, selected with INFRAHUB_BROKER_DRIVER=redis, so that a deployment can run without RabbitMQ. The driver requires Redis 6.2 or later. It connects to a single Redis server, or, through the new INFRAHUB_BROKER_URL setting, to a Redis server that Redis Sentinel manages.

Not decided yet:

  • Whether the driver is ready to merge. The branch still contains temporary changes that run all of CI on the Redis driver, and they must be reverted before merge.
  • What to do about the problems listed under "Problems found in review and not fixed yet".

How this came together

The commit hashes below are those on the branch after its rebase onto develop on 2026-11-11. Each rebase changes them.

  • The work started as a NATS → Redis conversion written with Claude Opus 4.5 (4309c5e0c0, authored on 2025-12-08), loosely modeled on PrefectHQ/prefect-redis.
  • Claude Fable 5 then took over and ran a systematic comparison against prefect-redis to find the gaps the conversion had missed. That surfaced and fixed: a message-loss race on group-less XREAD $ reads (73615a840e), broken event broadcast (a shared consumer group delivered each event to only one worker), refresh.settings.* misrouted to the work queue, silently dropped failure retries, missing crash recovery, unbounded stream growth, and the missing Redis default port (fa586f418e).
  • A second comparison against ag2ai/faststream's Redis package produced two more hardening commits: a catch-all backoff so consumer tasks survive unexpected errors (4ba34f7d94), and RPC timeout enforcement, because a lost reply left callers awaiting a future forever. The timeout originally covered all three broker adapters. During the rebase onto develop, develop's own RPC timeout replaced it for RabbitMQ, develop had already removed NATS, and the Redis part is now bc4832e450.
  • Three rounds of PR review hardening followed, replacing in-memory delayed retries with a durable delayed queue (62fdd543b8), isolating and dead-lettering undeliverable entries (37e8216e89, 24afadd3ce), making reply retention consumption-aware instead of cap-based (b4f02fe89b, 502d900290, a07c529bc2), and deduplicating the handler scaffolding (6a84ec50a7).
  • Design decisions (broadcast mechanism, retry semantics, idle-claim threshold, trimming strategy, deferred concurrency) were pressure-tested and confirmed interactively before implementation.

How the Redis driver delivers messages

The code is in redis.py.

  • Broadcast events (refresh.registry.*, refresh.settings.*, refresh.git.*) go to the stream {namespace}:events. Every Infrahub process reads the whole stream with XREAD and keeps only the routing keys it subscribes to. The stream is capped at about 10,000 entries.
  • Work items go to the stream {namespace}:rpcs, which the task workers share through the consumer group git-workers. A worker marks the work items it is handling as in progress every 15 seconds, and every 30 seconds each worker claims with XAUTOCLAIM the work items nobody has marked for a minute: those of a worker that stopped, and deliveries that never reached their worker. A worker that shuts down makes the work items it was handling claimable at once, so the next claim pass of another worker takes them over. A claimed work item delivered more than 5 times is moved to the list {namespace}:rpcs:dead, which keeps the newest 1,000 entries, instead of being run again. The stream is trimmed up to the oldest entry still pending. Every 30 seconds, a worker also deletes the group's consumers that hold no pending entry and have read nothing for 30 minutes, which belong to workers that stopped without their shutdown cleanup.
  • RPC replies go to one stream per process, {namespace}:callback:<worker id>. The process trims the entries it has read and keeps renewing the stream's 30-minute expiry, so the stream of a process that stopped without cleaning up deletes itself.
  • Each entry is handled in a task of its own, as the RabbitMQ client library does for each delivery. A task worker reads the work queue only while fewer than INFRAHUB_BROKER_MAXIMUM_CONCURRENT_MESSAGES of its work items are running, so the entries it cannot start yet stay with the group for the other workers. Events and RPC replies have no limit, as on RabbitMQ.
  • A failed work item is retried after a delay. The retry is stored in the sorted set {namespace}:delayed before the failed entry is acknowledged, so it survives a worker restart. A delayed entry that cannot be delivered is tried again every 60 seconds, and after 5 attempts it is moved to the list {namespace}:delayed:dead, which keeps the newest 1,000 entries.
  • A failed broadcast event is not retried. The driver drops a delayed message bound for the events stream and logs a warning, because every process would receive the retry and each process that failed it would publish another one. RabbitMQ drops these retries too, because its dead-letter exchange routes only work-queue keys.
  • rpc() waits for the timeout argument or INFRAHUB_BROKER_RPC_TIMEOUT seconds (120 by default), then raises WorkerTimeoutError, as the RabbitMQ driver does since a12d518b2c.

How the Redis driver connects to Redis

The code is in redis_connection.py and config.py.

  • INFRAHUB_BROKER_URL accepts the same redis://, rediss://, redis+sentinel:// and rediss+sentinel:// URLs as INFRAHUB_CACHE_URL. With a Sentinel URL, the driver asks the Sentinel nodes for the current master and follows a failover. The settings fail to load, so Infrahub does not start, when the URL is set and INFRAHUB_BROKER_DRIVER is not redis.
  • Without a URL, the driver connects to INFRAHUB_BROKER_ADDRESS on port 6379 (or INFRAHUB_BROKER_PORT), database 0, with INFRAHUB_BROKER_TLS_ENABLED, INFRAHUB_BROKER_TLS_INSECURE and INFRAHUB_BROKER_TLS_CA_FILE. INFRAHUB_BROKER_USERNAME and INFRAHUB_BROKER_PASSWORD default to the RabbitMQ default user infrahub/infrahub, and the driver does not send a username or password that is left at that default.
  • The driver builds its client with the same code as the cache, so it retries a command up to 10 times after a connection error, and checks a connection that has been idle for 20 seconds before using it.

Behavior during a Redis Sentinel failover

test_redis_sentinel.py starts one Redis master, one replica and one Sentinel, and sends RPCs from an API server bus to a task worker bus.

  • The tests check that an RPC is answered through Sentinel, after the master is killed, and after a failover requested with SENTINEL FAILOVER once Sentinel has demoted the old master.
  • In a diagnostic run of the requested failover, Sentinel demoted the old master about 10 seconds after promoting the replica, and until then the driver's connections stayed on the old master. Writes that the old master accepts in that window are lost when it becomes a replica of the new master, because Redis replicates asynchronously (Redis Sentinel documentation, section "Consistency under partitions").
  • In the same run, after the demotion, the connections that send write commands, XREADGROUP included, moved to the new master. The connections that only read with XREAD stayed on the old master, now a replica, and still received the RPC reply through replication.

Changes made while rebasing onto develop

The branch was rebased from stable onto develop on 2026-11-04, and the pull request now targets develop. On 2026-11-11, it was rebased again onto develop at 33100c4c1b.

  • develop removed the NATS message bus driver, so this branch no longer changes it.
  • develop added its own RPC timeout in a12d518b2c. The RabbitMQ and local drivers keep that code, and the Redis driver's rpc() follows the same rules (bc4832e450). The branch's RabbitMQ RPC timeout integration test was dropped, because test_rpc_timeout.py on develop covers the same behavior.
  • develop upgraded redis to 8.1.0 and enables the ruff rule against catching Exception without a reason. f38a911e30 adapts the driver to both, and fixes the TLS settings, which the client ignored before: it never loaded INFRAHUB_BROKER_TLS_CA_FILE and still verified the certificate when INFRAHUB_BROKER_TLS_INSECURE was set.
  • develop added INFRAHUB_CACHE_MAX_CONNECTIONS in f29e3a59b7, which sets the size of the cache and lock connection pools. 415e7783de keeps that size for the cache client, and the configuration still warns when a max_connections parameter in INFRAHUB_CACHE_URL overrides it. The message bus client is built without the setting, so its pool keeps the redis-py default of 100 connections.

Changes outside the driver

  • redis_connection.py: the cache and the broker share the code that builds a single-node URL, and build_broker_connection_string() and build_broker_redis_connection() are new (415e7783de).
  • config.py: BrokerSettings.url is new, the username and password settings have descriptions, and the global tls.ca_bundle fills broker.tls_ca_file when a broker URL is set, as it does for the cache URL.
  • Generated files: the configuration reference and the environment block of the root docker-compose.yml, which now lists INFRAHUB_BROKER_URL.
  • message_bus/__init__.py: the RPC publish-and-wait logic and the request id enricher, which the RabbitMQ and Redis drivers both used to carry, are written once here. The new listen(topic, listener) method passes the messages matching an AMQP topic to a listener, which infrahub events listen now uses with either driver (4949485e0d).
  • The knowledge doc message-bus.md and the changelog fragment +redis-message-bus.added.md.

Temporary changes that run CI on the Redis driver (revert before merge)

  • ci.yml sets INFRAHUB_BROKER_DRIVER=redis for every job. E2E-testing-invoke-demo-start stays on RabbitMQ, because it starts a released Infrahub image, which has no Redis driver.
  • version-upgrade.yml stays on RabbitMQ, because it starts Infrahub 1.8.0 before upgrading.
  • development/docker-compose.yml defaults the driver to redis and the broker address to cache.
  • The infrahub-testcontainers compose file, its cluster variant and container.py default the driver to redis and the broker address to cache (05a2032eb7, 92bfbf95cb):
    • The RabbitMQ service message-queue belongs to the compose profile rabbitmq, so CI stacks no longer start it.
    • With INFRAHUB_BROKER_DRIVER=rabbitmq, the env file turns on that profile and points the broker address at message-queue.
    • infrahub-server depends on message-queue with required: false, which needs Docker Compose 2.20 or later.
  • The RabbitMQ integration tests skip when the driver is not rabbitmq.

Problems found in review and fixed

  • 2901215fa1: a failed broadcast event was retried onto the events stream, so the number of runs grew with every retry round; a stream maintenance command that the server kept refusing stopped the consumer from reading its stream; and the default RabbitMQ credentials were sent to Redis, which refused a Redis server without that user.
  • 92bfbf95cb: the testcontainers stacks could start RabbitMQ but could not log in to it, because the compose files set empty broker credentials.
  • 4949485e0d:
    • Each stream consumer ran one handler at a time, so a long git flow delayed the events and work items behind it, and INFRAHUB_BROKER_MAXIMUM_CONCURRENT_MESSAGES had no effect.
    • The consumers of workers that stopped without a clean shutdown stayed in the git-workers group.
    • On Redis 6.2, EXPIRE ... NX failed after each RPC reply was written, and the required Redis version was not documented.
    • infrahub events listen only read from RabbitMQ.
    • Two comments in redis.py described behavior the code does not have, several policy values were literals, and the RPC wait and header building were written twice.
  • 549e9eb7b7:
    • A work item whose worker stopped before acknowledging it, or whose XREADGROUP reply was lost, was delivered again only after 30 minutes. It is now taken over once nobody has marked it as in progress for a minute, and within 30 seconds when its worker shut down.
    • A work item whose handler kept stopping its worker was passed from worker to worker without a limit. It now goes to the dead-letter list after 5 deliveries. The RabbitMQ driver has no delivery limit with its default classic queues, which do not support poison message handling (RabbitMQ documentation, "Classic Queue Features"); quorum queues drop or dead-letter a message after 20 deliveries by default since RabbitMQ 4.0 (RabbitMQ documentation, "Poison Message Handling").

Problems found in review and not fixed yet

A code review of this pull request on 2026-11-05 reported the following. They are not fixed, and they have not been reproduced.

  • When the consumer group is missing and re-creating it keeps failing, the consumer retries without waiting. Re-creating the group at id 0 delivers again every entry still in the stream. This is left as it is for now.
  • The RabbitMQ integration tests also skip when INFRAHUB_USE_TEST_CONTAINERS=false, so they no longer run against a RabbitMQ that is already running.

How this was tested

uv run pytest backend/tests/unit/test_redis_connection.py backend/tests/unit/config backend/tests/unit/services/adapters/message_bus
uv run pytest backend/tests/integration/services/adapters/message_bus/test_redis.py backend/tests/integration/services/adapters/message_bus/test_redis_sentinel.py

create-dataset runs: same duration as develop, and Redis storage stays bounded

On 2026-11-11, the TestCreateDataset test of infrahub-private-tests ran locally on one machine (24 CPUs, 80 GB of memory) with the small profile. Each run used an enterprise image built from infrahub-private develop, the same way the create-dataset workflow builds it. The results were not uploaded.

  • Duration, from two runs of develop (33100c4c1b) with RabbitMQ and two of this branch (cd5465b536) with Redis:
    • In the pair where the stack stayed up until the computed-attribute flows stopped, the last of those flows started 1918 s (RabbitMQ) and 1917 s (Redis) after test_load_initial_data started.
    • Both runs of that pair started 5125 computed_attribute_process_transform flows and 2726 query-computed-attribute-transform-targets flows.
    • The data load alone varies between runs of the same image: 1155 s to 1676 s with RabbitMQ, and 1235 s to 1509 s over three runs with Redis.
  • Redis storage, from a third run of this branch that recorded the Redis keys every 30 seconds for 90 minutes:
    • The run covered the data load, 10 idle minutes, and a restart of every task worker and API server (one of each with SIGTERM, the other with SIGKILL).
    • The message bus keys never used more than 54 KB.
    • Only {namespace}:events kept growing, by about one 584-byte entry per minute. The stream is capped at about 10,000 entries (redis.py).
    • The two consumers that the restarted task workers left in the git-workers group were removed 30 minutes after the restart.

Found in these runs and not fixed:

  • A task worker does not shut the message bus down when it stops, because only the API server calls the services' shutdown (server.py). The consumer of a stopped task worker, and its callback stream if it has one, stay until the 30-minute cleanup.
  • While the handler of one work item never finishes, its worker keeps marking the item as in progress. {namespace}:rpcs then keeps every later entry, because it is trimmed only up to the oldest pending entry (_trim_worked_stream). This was not reproduced.
  • Two of the three Redis runs logged Redis error in consumer with TimeoutError: Timeout reading from cache:6379 a few seconds after a process started loading schemas. redis-py 8.1 uses a 5-second read timeout by default. The cause is not known.

Traces of the refresh.registry.branches broadcast: one producer span, one consumer span in each process

Since fbbccb7b2e, the driver opens a producer span when it publishes a message (publish_message) or an RPC reply (publish_reply), and a consumer span (on_callback, on_message) in each process that handles an entry, under the trace context read from the entry headers (_producer_span, _traced_message). Besides routing_key, the spans carry these OpenTelemetry messaging attributes:

  • messaging.system (redis), messaging.destination.name (the stream), messaging.message.id (the stream entry ID) and messaging.operation.type (send or process)
  • messaging.consumer.group.name on a work item
  • messaging.message.conversation_id on an RPC request and its reply

Before fbbccb7b2e, the spans had the kind INTERNAL and only the routing_key attribute, and sending a reply opened no span.

On 2026-11-12, a local development stack of fbbccb7b2e ran with the Redis driver, the Tempo and Grafana services of docker-compose-observability.yml, and INFRAHUB_TRACE_SAMPLE_RATIO=1.0. It ran one API server with 4 worker processes and 2 task workers. Alloy and cAdvisor were left out, because Docker could not mount /var/run/docker.sock into a container on that machine; they collect logs and metrics, not traces.

  • Schema update, with infrahubctl schema load of a schema with one node kind: in the trace of POST /api/schema/load, the API server process that handled the request has a publish_message producer span for refresh.registry.branches. The span has 6 on_callback consumer children, one in each process of the stack, the publishing process included.
  • Branch merge, with infrahubctl branch merge of a branch with one new node: in the trace of the GraphQL POST, the merge flow (Merge branch trace-merge-redis-kinds into main) on a task worker has the publish_message producer span. The span again has 6 on_callback consumer children, one in each process.
  • Tempo's service graph recorded messaging_system edges from infrahub-task-worker to infrahub-task-worker for these spans. With the INTERNAL spans of the earlier runs, it recorded none.
  • An RPC request and its reply are not part of these two operations. test_redis_rpc_is_traced_through_the_request_and_the_reply checks them against a Redis server: publish_message in the caller, on_message in the task worker as its child, publish_reply under on_message, and on_callback in the caller as the child of publish_reply.

In each pair of images below, the first one lists the result of a TraceQL query: the publish_message span whose parent is the request or the merge flow, and the on_callback spans whose parent is publish_message, with the kind, service, process id (worker.id), stream and entry ID of each span. The second one is the trace view filtered to the same spans, with the details of publish_message open. Grafana prints the service name on some rows of the trace view only; the first image lists the service of every span.

Schema update:

TraceQL result for the schema update: one producer publish_message span in infrahub-server and six consumer on_callback spans, in four infrahub-server and two infrahub-task-worker processes, all with the same stream and entry ID

Trace view of POST /api/schema/load filtered to publish_message and on_callback, with the publish_message details showing the kind producer, 6 children and the messaging attributes

Branch merge:

TraceQL result for the branch merge: one producer publish_message span in infrahub-task-worker and six consumer on_callback spans, in four infrahub-server and two infrahub-task-worker processes, all with the same stream and entry ID

Trace view of the branch merge filtered to publish_message and on_callback, with the publish_message details showing the kind producer, 6 children and the messaging attributes