"Could not return the broken resource to the pool" thrown from get due to failure to create an idle connection

Expected behavior

I have a test using Spring Data Redis and Jedis in using clustered mode. In this test I am killing and restarting redis servers. I expect that jedis will retry my get operation until it succeeds if a server is killed and restarted.

Actual behavior

What I see is that sometimes the client fails with the below exception. In the stack trace, it appears that it is failing to create an idle connection in the pool in (probably because the server has been killed).

Caused by: redis.clients.jedis.exceptions.JedisException: Could not return the broken resource to the pool
	at redis.clients.jedis.util.Pool.returnBrokenResourceObject(Pool.java:126)
	at redis.clients.jedis.util.Pool.returnBrokenResource(Pool.java:103)
	at redis.clients.jedis.Jedis.close(Jedis.java:4065)
	at redis.clients.jedis.JedisClusterCommand.releaseConnection(JedisClusterCommand.java:221)
	at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:149)
	at redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:69)
	at redis.clients.jedis.BinaryJedisCluster.get(BinaryJedisCluster.java:235)
	at org.springframework.data.redis.connection.jedis.JedisClusterStringCommands.get(JedisClusterStringCommands.java:65)
	... 21 more
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Failed to create socket.
	at redis.clients.jedis.DefaultJedisSocketFactory.createSocket(DefaultJedisSocketFactory.java:110)
	at redis.clients.jedis.Connection.connect(Connection.java:226)
	at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:140)
	at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:310)
	at redis.clients.jedis.BinaryJedis.initializeFromClientConfig(BinaryJedis.java:88)
	at redis.clients.jedis.BinaryJedis.<init>(BinaryJedis.java:293)
	at redis.clients.jedis.Jedis.<init>(Jedis.java:169)
	at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:177)
	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:565)
	at org.apache.commons.pool2.impl.GenericObjectPool.ensureIdle(GenericObjectPool.java:631)
	at org.apache.commons.pool2.impl.GenericObjectPool.invalidateObject(GenericObjectPool.java:1006)
	at org.apache.commons.pool2.impl.GenericObjectPool.invalidateObject(GenericObjectPool.java:976)
	at redis.clients.jedis.util.Pool.returnBrokenResourceObject(Pool.java:124)
	... 28 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:607)
	at redis.clients.jedis.DefaultJedisSocketFactory.createSocket(DefaultJedisSocketFactory.java:80)
	... 40 more

Steps to reproduce:

I’m afraid I don’t have a reproducible test case right now. This issue happens if:

1. A server is killed, causing an operation to fail in JedisClusterCommand.runWithRetries
2. There are multiple client threads doing operations and some are waiting to get objects from the pool, such that the pool’s hasTakeWaiters is true
3. The call to JedisClusterCommand.releaseConnection ends up trying to create a new idle connection in the pool, because hasTakeWaiters is true and there are not enough idle connections
4. Creating the idle connection fails with a connection exception (because the server is not running)
5. The connect exception bubbles up and is not caught in runWithRetries because it happened from returnConnection

Jedis version:

Jedis 3.7.0

[upthewaterspout]

@nikitaron - I haven’t created any patches to jedis to fix this issue. There is a workaround - use a new GenericObjectPoolConfig.setMaxTotal(-1). That will use an unbounded pool and then you’ll never have any threads waiting to get objects from the pool. The downside is that you might end up with a lot of connections in the pool if you have a lot of threads.

[sazzad16]

Closing in favor of #2745