Randomly getting "RedisCommandTimeoutException: Command timed out"

Bug Report

Current Behavior

We’re using Redis 3.2.5 with Spring Data Redis as a client. As part of the upgrade to Spring Boot 2, we switched from Jedis to Lettuce 5.0.4.RELEASE. Ever since the move we’ve been getting the following exceptions quite often, but at random times:

org.springframework.dao.QueryTimeoutException: Redis command timed out; nested exception is io.lettuce.core.RedisCommandTimeoutException: Command timed out
        at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:70) ~[spring-data-redis-2.0.7.RELEASE.jar!/:2.0.7.RELEASE]
        at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41) ~[spring-data-redis-2.0.7.RELEASE.jar!/:2.0.7.RELEASE]
        at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44) ~[spring-data-redis-2.0.7.RELEASE.jar!/:2.0.7.RELEASE]
        at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42) ~[spring-data-redis-2.0.7.RELEASE.jar!/:2.0.7.RELEASE]
        at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:257) ~[spring-data-redis-2.0.7.RELEASE.jar!/:2.0.7.RELEASE]
        at org.springframework.data.redis.connection.lettuce.LettuceHashCommands.convertLettuceAccessException(LettuceHashCommands.java:445) ~[spring-data-redis-2.0.7.RELEASE.jar!/:2.0.7.RELEASE]
        at org.springframework.data.redis.connection.lettuce.LettuceHashCommands.hMGet(LettuceHashCommands.java:321) ~[spring-data-redis-2.0.7.RELEASE.jar!/:2.0.7.RELEASE]
        at org.springframework.data.redis.connection.DefaultedRedisConnection.hMGet(DefaultedRedisConnection.java:895) ~[spring-data-redis-2.0.7.RELEASE.jar!/:2.0.7.RELEASE]
        at org.springframework.data.redis.connection.DefaultStringRedisConnection.hMGet(DefaultStringRedisConnection.java:483) ~[spring-data-redis-2.0.7.RELEASE.jar!/:2.0.7.RELEASE]
        at org.springframework.data.redis.core.DefaultHashOperations.lambda$multiGet$7(DefaultHashOperations.java:161) ~[spring-data-redis-2.0.7.RELEASE.jar!/:2.0.7.RELEASE]
        at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:224) ~[spring-data-redis-2.0.7.RELEASE.jar!/:2.0.7.RELEASE]
        at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:184) ~[spring-data-redis-2.0.7.RELEASE.jar!/:2.0.7.RELEASE]
        at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:95) ~[spring-data-redis-2.0.7.RELEASE.jar!/:2.0.7.RELEASE]
...
Caused by: io.lettuce.core.RedisCommandTimeoutException: Command timed out
	at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114) ~[lettuce-core-5.0.4.RELEASE.jar!/:na]
	at io.lettuce.core.FutureSyncInvocationHandler.handleInvocation(FutureSyncInvocationHandler.java:62) ~[lettuce-core-5.0.4.RELEASE.jar!/:na]
	at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80) ~[lettuce-core-5.0.4.RELEASE.jar!/:na]
	at com.sun.proxy.$Proxy152.hmget(Unknown Source) ~[na:na]
	at org.springframework.data.redis.connection.lettuce.LettuceHashCommands.hMGet(LettuceHashCommands.java:319) ~[spring-data-redis-2.0.7.RELEASE.jar!/:2.0.7.RELEASE]

Everything may work perfectly for a few days, and then we would see this. I should note that the application workload on Redis is very low, to the point where we might not even make a single call to Redis for a few hours.

We tried reproducing this issue in several ways but failed. We verified that there are no network issues connecting to Redis, and that Redis is always up. We also looked for similar issues - the only one which seemed relevant is https://github.com/lettuce-io/lettuce-core/issues/296, but that was resolved a long time ago.

Expected behavior/code

No timeouts should occur in our setting.

Environment

• Lettuce version(s): 5.0.4.RELEASE
• Redis version: 3.2.5
• Spring Boot version: 2.0.3.RELEASE
• Spring Data Redis version: 2.0.7.RELEASE

[Arch-vile]

To be clear here. I do not think there is any issue with the Lettuce. If application for some reason or another causes massive CPU load and goes to halt, Lettuce is not to blame to not able to handle this.

@mp911de, I have no motivation to open a new ticket. I have no questions on the matter. For me this is the most logical place to add my experience/solutions to the issue, as I imagine others end up here googling with the exception message and maybe my comments will be of value for them. If my logic is wrong here, please explain a bit more why opening a new ticket is preferred. Thanks!

[MatanRubin]

@Arch-vile , thank for the detailed info! We were trying everything to reproduce this issue but couldn’t - it happened only in production. It’s great that you found out how to reproduce it. We ended up going back to Jedis due to this issue. @mp911de, @Arch-vile’s answer here is valuable and in context. I expected you to thank him for his research and reopen the issue, as there’s clearly a real problem here.