feat(redis): Redis Sentinel (HA) support — app Redis + Celery, TLS-aware

Merged
JU
4mo ago
+397-10

Description

Adds Redis Sentinel support so Onyx can connect to a highly-available Redis (Sentinel discovers the current master and follows failover) instead of a fixed host. Requested by a customer running a Sentinel-fronted Redis.

Config — REDIS_SENTINEL_HOSTS (comma-separated host:port nodes) switches on Sentinel mode; REDIS_SENTINEL_MASTER_NAME (default mymaster); optional REDIS_SENTINEL_PASSWORD for the sentinel nodes (the master/replica still use REDIS_PASSWORD).

App Redis (redis_pool.py) — sync create_pool returns Sentinel(...).master_for(...).connection_pool (or slave_for for the replica), which plugs into the existing redis.Redis(connection_pool=...) flow unchanged; async _build_async_redis_connection uses redis.asyncio.sentinel. TLS (the native ssl_* kwargs from the mTLS work) is applied to both the sentinel-node connections and the master/replica data connections.

Celery (celery/configs/base.py) — sentinel:// URLs (one per node) for the broker and result backend, master_name in the transport options, and broker_use_ssl / redis_backend_use_ssl for TLS; sentinel-node auth + TLS go in sentinel_kwargs.

IAM auth and the direct (non-sentinel) path are unchanged.

Stacked on #12342 (async Redis TLS fix) — built on that branch so the Sentinel TLS path uses the corrected native-kwargs async mechanism. Rebase onto main once #12342 merges.

How Has This Been Tested?

  • Unit tests (test_redis_sentinel.py): sync create_pool routes through Sentinel.master_for (and slave_for for the replica); async routes through redis.asyncio.sentinel; TLS + auth apply to both the sentinel and data connection sets; the Celery broker + result backend render sentinel:// URLs with master_name. The existing direct-path tests still pass.
  • Live validation status: the app-Redis routing is unit-tested against the real redis-py Sentinel API (mocked transport). A full live failover test against a running master/replica/sentinel cluster — and live validation of the Celery sentinel:// + broker_use_ssl path (kombu has some quirks here) — is recommended before relying on it in production; I can add that as a follow-up.

Additional Options

  • [Optional] Please cherry-pick this PR to the latest release version.
  • [Optional] Override Linear Check

Summary by cubic

Adds Redis Sentinel HA support for app Redis (sync + async) and Celery with full TLS/mTLS. Also adds strict config validation, blocks Sentinel+IAM, and fixes Celery Sentinel TLS negotiation.

  • New Features

    • HA via REDIS_SENTINEL_HOSTS with REDIS_SENTINEL_MASTER_NAME (default mymaster) and optional REDIS_SENTINEL_PASSWORD.
    • App Redis uses redis-py Sentinel for master/replica discovery and failover; TLS applies to both sentinel-node and data connections.
    • Celery broker/result backend use sentinel:// URLs with master_name; TLS via broker_use_ssl/redis_backend_use_ssl; sentinel_kwargs carries sentinel auth/TLS.

  • Bug Fixes

    • Async Redis TLS uses native ssl_* kwargs; async IAM auth no longer crashes on ssl_context.
    • Celery Sentinel TLS sets ssl=True in sentinel_kwargs; result_backend_transport_options is always defined.
    • Config validation: REDIS_SENTINEL_HOSTS entries must be valid host:port with port 1–65535; set-but-empty or malformed raises; empty/whitespace REDIS_SENTINEL_MASTER_NAME is rejected; Sentinel cannot be combined with USE_REDIS_IAM_AUTH.