Help，pls，side effects of disabling Jedis retries during Redis Cluster failover to prevent thread blocking

[heshuhua]

When a Redis cluster experiences a master node failure while connected with Jedis, there is a delay before a replica is promoted to master. Under high traffic, this causes Jedis client threads to block. We observed that disabling Jedis retries mitigates the performance impact on other Redis shards. What are the potential side effects of this approach? Furthermore, what would be the best practice to handle this situation?

[ggivo]

@heshuhua
Hi,

Trying to provide some hints, assuming you are referring to using RedisClusterClient (formerly JedisCluster).

We observed that disabling Jedis retries mitigates the performance impact on other Redis shards. What are the potential side effects of this approach?

RedisClusterClient handles both redirect following (on ASK/MOVED) and retries on errors based on the same configuration setting, maxAttempts. Disabling retries completely also means that redirects will no longer be followed.

We observed that disabling Jedis retries mitigates the performance impact on other Redis shards.

Currently, Jedis attempts a topology refresh (e.g., refreshing slot/node cluster info) on a redirect or after two sequential connection errors. See here. and here.
One possible reason (this would need to be confirmed by examining stack traces of blocked threads) is that the observed impact on other shards is caused by the topology refresh operation. The topology refresh is synchronous and will block other threads using the client until it completes.

If this root cause is confirmed, it might make sense to introduce a grace period after a topology refresh is performed.