Unable to acquire subscription lock

During relatively high load (~150-250 requests per minute), I often see this error

Caused by: org.redisson.client.RedisTimeoutException: Unable to acquire subscription lock after 3000ms. Increase 'subscriptionsPerConnection' and/or 'subscriptionConnectionPoolSize' parameters.
	at org.redisson.pubsub.PublishSubscribe.lambda$subscribe$2(PublishSubscribe.java:69)
	at io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:715)
	at io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)
	at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:703)
	at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:790)
	at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:503)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)

I doubled the subscriptionsPerConnection from 5 to 10, the subscriptionConnectionPoolSize from 50 to 100, and the connectionPoolSize from 64 to 128, but it doesn’t seem to have helped.

How can I go about diagnosing/debugging this? Do I need to just keep increasing these parameters till it works?

Redis version

6.x

Redisson version

3.16.8

Redisson configuration

config.useSingleServer().apply {
    path.toIntOrNull()?.also {
        database = it
    } ?: run {
        logger.warn { "Unable to parse database number: $path from redis configuration" }
    }
    
    subscriptionsPerConnection = 10
    subscriptionConnectionPoolSize = 100
    connectionPoolSize = 128
    timeout = 6000
}
config.nettyThreads = 64

I have also tried using Redisson versions 12.0, 13.0, 14.0, 15.0, 16.0 and 17.0 without luck.

[Sovietaced]

Please try 3.17.4 version, since it has fix for #4294

We have been experiencing this issue for years. I had raised a similar thread a while ago. It started happening to us again recently so I decided to search again and found this thread.

For us the indication that something was very obviously broken on the client side is that the operation would timeout immediately despite claiming that the timeout was many seconds.

2022-07-29T17:56:07.021Z - Acquiring lock for fleet-info-cache:fleet-info-sync-task task… 2022-07-29T17:56:07.021Z - org.redisson.client.RedisTimeoutException: Subscribe timeout: (7500ms). Increase ‘subscriptionsPerConnection’ and/or ‘subscriptionConnectionPoolSize’ …

We had very similar issues where seemingly only some of our kubernetes pods would experience this issue. It would remain throughout reboots.

So I took this opportunity to try upgrading to 3.17.5 and it appears to have solved the problem!

[mrniko]

@Sovietaced

Thanks for feedback!