Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Slow gets from redisson in CommandServiceAsyncGet

See original GitHub issue

Hi,

We are using redisson-client 3.6.0 with cluster mode enabled with elasticache, redis=3.2.6(encryption-at-rest enabled). These are the configurations for 2 separate clusters,

{
  "clusterServersConfig": {
    "idleConnectionTimeout": 60000,
    "pingTimeout": 1000,
    "connectTimeout": 10000,
    "timeout": 5000,
    "retryAttempts": 3,
    "retryInterval": 1500,
    "reconnectionTimeout": 3000,
    "failedAttempts": 3,
    "subscriptionsPerConnection": 5,
    "clientName": "RegularCluster",
    "sslEnableEndpointIdentification": true,
    "sslProvider": "JDK",
    "pingConnectionInterval": 0,
    "keepAlive": false,
    "tcpNoDelay": false,
    "loadBalancer": {
      "class": "org.redisson.connection.balancer.RoundRobinLoadBalancer"
    },
    "slaveConnectionMinimumIdleSize": 50,
    "slaveConnectionPoolSize": 150,
    "masterConnectionMinimumIdleSize": 50,
    "masterConnectionPoolSize": 150,
    "readMode": "MASTER",
    "subscriptionMode": "SLAVE",
    "subscriptionConnectionMinimumIdleSize": 1,
    "subscriptionConnectionPoolSize": 50,
    "dnsMonitoringInterval": 5000,
    "nodeAddresses": [
      "redis://elastic-cache-config-end-point:6379"
    ],
    "scanInterval": 5000,
    "slaveSubscriptionConnectionPoolSize": 50,
    "slaveSubscriptionConnectionMinimumIdleSize": 1
  },
  "threads": 25,
  "nettyThreads": 50,
  "codec": {
    "class": "org.redisson.codec.SnappyCodec"
  },
  "referenceCodecProvider": {
    "class": "org.redisson.codec.DefaultReferenceCodecProvider"
  },
  "referenceEnabled": true,
  "transportMode": "NIO",
  "lockWatchdogTimeout": 30000,
  "keepPubSubOrder": true,
  "useLinuxNativeEpoll": false
}

{
  "clusterServersConfig": {
    "idleConnectionTimeout": 60000,
    "pingTimeout": 1000,
    "connectTimeout": 10000,
    "timeout": 5000,
    "retryAttempts": 3,
    "retryInterval": 1500,
    "reconnectionTimeout": 3000,
    "failedAttempts": 3,
    "subscriptionsPerConnection": 5,
    "clientName": "HighAvailCluster",
    "sslEnableEndpointIdentification": true,
    "sslProvider": "JDK",
    "pingConnectionInterval": 0,
    "keepAlive": false,
    "tcpNoDelay": false,
    "loadBalancer": {
      "class": "org.redisson.connection.balancer.RoundRobinLoadBalancer"
    },
    "slaveConnectionMinimumIdleSize": 50,
    "slaveConnectionPoolSize": 150,
    "masterConnectionMinimumIdleSize": 50,
    "masterConnectionPoolSize": 150,
    "readMode": "MASTER",
    "subscriptionMode": "SLAVE",
    "subscriptionConnectionMinimumIdleSize": 1,
    "subscriptionConnectionPoolSize": 50,
    "dnsMonitoringInterval": 5000,
    "nodeAddresses": [
      "redis://elastic-cache-config-end-point:6379"
    ],
    "scanInterval": 5000,
    "slaveSubscriptionConnectionPoolSize": 50,
    "slaveSubscriptionConnectionMinimumIdleSize": 1
  },
  "threads": 25,
  "nettyThreads": 50,
  "codec": {
    "class": "org.redisson.codec.SnappyCodec"
  },
  "referenceCodecProvider": {
    "class": "org.redisson.codec.DefaultReferenceCodecProvider"
  },
  "referenceEnabled": true,
  "transportMode": "NIO",
  "lockWatchdogTimeout": 30000,
  "keepPubSubOrder": true,
  "useLinuxNativeEpoll": false
}

We see a degraded performance on all our apis and the bottle-neck seems to be from redisson. We are using spring caching and noticed that all the keys with a particular cacheName are being stored in a single shard instead of being distributed. How do we solve for actually distributing this across different shards?

We are contemplating switching to redisson.pro if it’s going to help. But does the configuration look good and will redisson.pro help with the issues?

Please find attached screenshots that is showing the bottle-neck

Issue Analytics

State:
Created 6 years ago
Comments:9 (4 by maintainers)

Top GitHub Comments

2reactions

mrnikocommented, Feb 3, 2018

@PC-AKumar

What seems to be helping right now is going back to performance HIGHER_THROUGHPUT performance mode and increasing the master connection pool size to 300 with minimum idle and max idle at 60

What is the average size of objects you’re stored into spring cache?

I liked the near cache concept of org.redisson.spring.cache.RedissonSpringLocalCachedCacheManager but I am unsure if it will support clustered sharding. If it does this will add a significant performance boost for us.

I have just released 3.6.0-3 version it contains org.redisson.spring.cache.RedissonSpringClusteredLocalCachedCacheManager class. It supports local cache and works in cluster.

0reactions

PC-AKumarcommented, Feb 5, 2018

@mrniko We store roughly 17kb on an average, median will probably around 50

Thanks for the update with near cache, will try it out.

Top Results From Across the Web

Why Redis get in async function is too slow? - node.js

Using await basically means "wait for this instruction to be completed before you do anything else". Not using it would make your script ......

Timeout Exception but no slow query in slowlog on Redis ...

When we look in Redis server's slowlog (tracks all query >10ms), there is no entry.When reducing the slowlog config value, I get some...

Optimize Redis Client Performance for Amazon ElastiCache ...

Buffering a sequence of commands in memory and sending them to the Redis server as a single batch. We refer to this method...

High-Concurrency Practices of Redis: Snap-Up System

If all queries are processed slowly, each service thread retrieves a new persistent connection, which will be consumed gradually. If this is the ......

Redis persistence

AOF can be slower than RDB depending on the exact fsync policy. ... From now on, every time Redis receives a command that...