question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

lettuce cluster multiGet RedisCommandTimeoutException cause memory leak

See original GitHub issue

I used lettuce 5.0.5.RELEASE and Spring Boot 2.0

and I write the code to multiGet value

  @Autowired
   private RedisTemplate<String, byte[]> redisTemplate;

        redisTemplate.opsForValue().multiGet(
            ids.stream()
                .map(RedisVideoProfileCache::genCacheKey)
                .collect(Collectors.toSet())
        )

Whenever I start the service for a long time, some app’s memory is leaked I download a jmap to debug it

 num     #instances         #bytes  class name
----------------------------------------------
   1:       5668369     3397294608  [B
   2:       9985354      455769512  [Ljava.lang.Object;
   3:       9797176      313509632  java.util.HashMap$Node
   4:       9700404      232809696  java.util.concurrent.CompletableFuture
   5:       8891675      213400200  java.util.ArrayList
   6:       3189943      127597720  io.lettuce.core.output.KeyValueListOutput
   7:       2802968      112118720  java.util.concurrent.CompletableFuture$BiRelay
   8:       3190015      102080480  io.lettuce.core.protocol.AsyncCommand
   9:       3190015      102080480  io.lettuce.core.protocol.Command
  10:       2358668       94346720  java.util.concurrent.CompletableFuture$UniWhenComplete
  11:       1077001       92868160  [C
  12:       3218728       77249472  io.lettuce.core.protocol.CommandArgs$KeyArgument
  13:       2399019       76768608  java.util.ArrayList$Itr
  14:       3190015       76560360  io.lettuce.core.protocol.CommandArgs
  15:        107450       76403064  [Ljava.util.HashMap$Node;
  16:       1572519       62900760  java.util.concurrent.CompletableFuture$UniApply
  17:       2411845       57884280  io.lettuce.core.KeyValue
  18:       3224435       51590960  java.lang.Integer
  19:        801185       32047400  io.lettuce.core.cluster.ClusterCommand
  20:        786213       31448520  java.util.concurrent.CompletableFuture$UniHandle
  21:       1251582       30037968  java.util.concurrent.CompletableFuture$CoCompletion
  22:        225886       29931880  [I
  23:        457000       25592000  org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation
  24:        786283       25161056  io.lettuce.core.DefaultConnectionFuture
  25:        826014       23765088  [Ljava.util.concurrent.CompletableFuture;
  26:        853506       20484144  java.lang.String
  27:        786652       18879648  io.lettuce.core.cluster.PooledClusterConnectionProvider$$Lambda$1028/1326396018
  28:        786652       18879648  io.lettuce.core.cluster.PooledClusterConnectionProvider$$Lambda$1030/1117636922
  29:        786652       18879648  io.lettuce.core.cluster.PooledClusterConnectionProvider$$Lambda$1031/481259366

the byte[] used large than normal case

When I checked this app error log, I found large count RedisCommandTimeoutException

 less c67003c59bbb854f176124c48d4c172a0d7aff1b1d14725705a7d3e57aad9e4a.log  | grep ERROR | grep  "io.lettuce.core.RedisCommandTimeoutException: Command timed out" | wc
  12804  320100 3111372

less c67003c59bbb854f176124c48d4c172a0d7aff1b1d14725705a7d3e57aad9e4a.log  | grep ERROR | grep -v "io.lettuce.core.RedisCommandTimeoutException: Command timed out" | wc
     20     467    4835

I suspect this exception caused a memory leak, why ?


Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:19 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
mp911decommented, Sep 25, 2018

Watch out for the channel handlers (in particular CommandHandler) and the attached DefaultEndpoint. It’s interesting to inspect the state of the connection, whether it’s connected and how many instances of connections you have. If you don’t have any concerns regarding data privacy issues with your heap dump, feel free to share it with me.

0reactions
shunminlicommented, Sep 30, 2018

Upgrade lettuce version to 5.1.0.RELEASE fixed.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Memory leak Error when connect redis with lettuce client
This type of error typically occurs when multiple instances of RedisClient get created without sharing ClientResources .
Read more >
lettuce-io/Lobby - Gitter
I have a problem getting a connection for Cluster working. ... I'm trying to identify the root cause of a memory leak in...
Read more >
Lettuce Reference Guide
Reactor based applications can sustain very high throughput message rates and operate with a very low memory footprint, making it suitable ...
Read more >
[Java] New Relic agent leaking memory when Lettuce Redis ...
Newrelic agent is leaking memory in our java app which is using redis reactive library(Lettuce) and became major blocker to go 100 %...
Read more >
Memory leak in ray head
CPU and memory usage on ray-head pod is still increasing and has to be restarted ... objects in cluster but it is probably...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found