RedisCommandTimeoutException: Command timed out after 1 second(s)

Bug Report

Current Behavior

Stack trace

2020-07-23 22:29:50,920 [http-nio2-8080-exec-128] ERROR com.paytm.pgplus.redis.RedisBridgeWorker.get():94 {REQUEST_ID=1595523588463} - Exception while reading from redis cluster: 
io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 second(s)
	at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
	at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:114)
	at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
	at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
	at com.sun.proxy.$Proxy211.get(Unknown Source)

Input Code

Input Code

// Code to configure redis client and establish a connection with redis:

private final SerializedObjectCodec serializedObjectCodec = new SerializedObjectCodec();
    private static Long defaultExpireTimeInSecond;
    private RedisClusterClient clusterClient = null;
    private StatefulRedisClusterConnection<String, Object> clusterConnection;

@PostConstruct
    private void connectToCluster() {

        RedisURI[] nodes = redisClusterClientUtil.getRedisURINodes(addresses);
        clusterClient = RedisClusterClient.create(Arrays.asList(nodes));
        setClusterClientOptions(clusterClient);

        clusterConnection = clusterClient.connect(serializedObjectCodec);
    }

    @PreDestroy
    public void closeConnection() {
        LOGGER.info("closing cluster connection");
        clusterConnection.close();
        clusterClient.shutdown();
    }

// Code to fetch connection at the time of operation calling(get, set,etc):

private StatefulRedisClusterConnection<String, Object> getConnection() {
        try {
            if (clusterConnection == null) {
                synchronized (RedisClusterClientLettuceService.class) {
                    if (clusterConnection == null) {
                        LOGGER.info("creating connection");
                        clusterConnection = clusterClient.connect(serializedObjectCodec);
                    }
                }
            }
            return clusterConnection;
        } catch (Exception e) {
            LOGGER.error("Failed to get connection!");
        }
        throw new RuntimeException("Failed to get connection");
    }

Properties used to configure lettuce:

redis.cluster.topologyRefreshOptions.enable=true redis.cluster.cancelCommandsOnReconnectFailure=true redis.cluster.enablePeriodicRefresh=true redis.cluster.periodicRefreshPeriod.seconds=30(in seconds) redis.cluster.enableAllAdaptiveRefreshTriggers=true redis.cluster.timeout.millis=1000(in ms) redis.cluster.refreshTriggerReconnectAttempts=5

Expected behavior/code

Environment

• Lettuce version(s): 5.2.2.RELEASE
• Redis version: 5.0.7
• We have a config of 5 Master- 5 slave configuration

Possible Solution

Additional context

• This issue is not consistent across all the application servers, the issue is intermittent and due to this a CPU spike is also observed.

• As soon as we boot up the server in production and the high load traffic hits the server, we start getting this issue

• Also, we have recently started facing this issue and the system was up from last 3 months, and never faced this issue

[trajano]

Thanks @kiteplayer your suggestion about the switching of the spring-cloud dependencies didn’t work. I was on Hoxton.SR8 and I switched to SR6. However, your other suggestion of switching to jedis worked out well for me. In which case I did this

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-data-redis</artifactId>
      <exclusions>
        <exclusion>
          <groupId>io.lettuce</groupId>
          <artifactId>lettuce-core</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>redis.clients</groupId>
      <artifactId>jedis</artifactId>
      <scope>runtime</scope>
    </dependency>

I am still wondering why on one project the default which is lettuce worked but on another it didn’t. I don’t see anything particularly different between the two projects in terms of redis. Also I am using Redis in a container environment and it is isolated so it’s an empty redis on both when it starts.

[kiteplayer]

Dear friend，     I am kiteplayer,Very glad to hear from you. I am not sure whether this problem is  solved completely,because I use the SpringBoot framework integrate springcloud and springcloud Aliabba， when I change the version of springcloud and springcloud Aliabba，this issue disappeared。But i still don’t know the root cause，I am sorry。 This follows are my program version,FYI. or you also try to use jedis replace lettuce. <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.3.0.RELEASE</version> <relativePath/> </parent>

<dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-dependencies</artifactId> <version>Hoxton.SR6</version> <type>pom</type> <scope>import</scope> </dependency>

    <dependency&gt;
        <groupId&gt;com.alibaba.cloud</groupId&gt;
        <artifactId&gt;spring-cloud-alibaba-dependencies</artifactId&gt;
        <version&gt;2.1.0.RELEASE</version&gt;
        <type&gt;pom</type&gt;
        <scope&gt;import</scope&gt;
    </dependency&gt;
</dependencies&gt;

</dependencyManagement>

------------------ 原始邮件 ------------------ 发件人: “lettuce-io/lettuce-core” <[e-mail]>; 发送时间: 2020年9月26日(星期六) 下午4:47 收件人: “lettuce-io/lettuce-core”<[e-mail]>; 抄送: “青山流水”<[e-mail]>;“Mention”<[e-mail]>; 主题: Re: [lettuce-io/lettuce-core] RedisCommandTimeoutException: Command timed out after 1 second(s) (#1362)

hello @Srthak100 May I ask How did you solve this problem in the end? I get the same situation in the env Lettuce version(s): 5.2.2.RELEASE Redis version: 5.0.7

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.