fix(celery): resolve the broker client through Sentinel when Sentinel is on

Merged
JU
3d ago
+61-0

Description

With REDIS_SENTINEL_HOSTS set, Celery's broker_url is a kombu sentinel:// URL (configs/base.py). celery_get_broker_client passed that URL to Redis.from_url. redis-py only accepts redis://, rediss:// and unix://, so every caller raised:

ValueError: Redis URL must specify one of the following schemes (redis://, rediss://, unix://)

This breaks every task that reads Celery queue depth on Sentinel deployments:

  • the user file processing beat check (the safety net that re-queues files stuck in PROCESSING; upload-time enqueue goes through kombu and still works)
  • user file delete and project sync
  • pruning
  • connector deletion
  • doc permission sync and external group sync
  • the monitoring tasks

The bug is present in every release since Sentinel support (#12347).

The fix: when REDIS_SENTINEL_HOSTS is set and the broker URL is sentinel://, build the client from RedisPool.create_pool on REDIS_DB_NUMBER_CELERY. This reuses the Sentinel auth, TLS, timeout and retry settings that the app Redis pool already uses.

All other URLs keep the Redis.from_url path. The URL check also covers one odd setup: Sentinel hosts set, plus a CELERY_BROKER_URL override that points at plain Redis. That setup keeps the URL it was given.

How Has This Been Tested?

  • Unit tests in test_celery_redis.py:
    • A Sentinel URL resolves through RedisPool.create_pool(db=REDIS_DB_NUMBER_CELERY, replica=False) and never calls from_url.
    • A redis:// URL still uses from_url when Sentinel hosts are set.
  • Live check: a local redis:7 master plus a redis:7 Sentinel, with REDIS_SENTINEL_HOSTS set and a Celery app built from configs/base.py.
    • Broker URL: sentinel://127.0.0.1:26391/15.
    • The old Redis.from_url path raises the error above.
    • The new client pings the master, uses DB 15, returns the correct celery_get_queue_length, and the second call reuses the cached client.

Additional Options

  • [Optional] Please cherry-pick this PR to the latest release version.

🤖 Generated with Claude Code

Summary by cubic

Fixes celery_get_broker_client crashing with ValueError when Celery's broker URL is a kombu sentinel:// URL. This broke every task that reads Celery queue depth on Sentinel deployments, blocking user file processing, pruning, connector deletion, permission sync, and monitoring.

When REDIS_SENTINEL_HOSTS is set and the broker URL starts with sentinel://, the client is now built from RedisPool.create_pool on REDIS_DB_NUMBER_CELERY, reusing the Sentinel auth, TLS, and timeout settings of the app Redis pool. All other URLs keep the Redis.from_url path.

Summary by CodeRabbit

  • New Features

    • Added support for Redis Sentinel broker connections, using configured Sentinel hosts, Celery database selection, and connection limits.
    • Preserved support for standard Redis broker URLs when Sentinel configuration is not applicable, ensuring existing connection behavior continues to work.

  • Tests

    • Added coverage for Sentinel and non-Sentinel broker connection behavior, including verification of the appropriate connection method in each configuration.