Why is Redisson not writing data to some shards in AWS ElactiCache (cluster mode enabled)?

Hi! I am using Redisson 3.16.3 with AWS ElastiCache in cluster mode enabled.

AWS ElastiCache Setup:

• Stage#1: Redis in cluster mode enabled, with 2 shards and each shard having 1 primary and 2 replica nodes.
• Stage#2: After some time, I added 2 more shards to this cluster, thus it has a total of 4 shards (4 primaries + 8 replicas).
• Stage#3: Eventually, I removed one cluster, thus the final setup is 3 shards (3 primaries + 6 replicas).

Application Code:

// Redisson Config
val config = new Config().setCodec(codec)
val clusterConfig = config.useClusterServers()
clusterConfig.setMasterConnectionPoolSize(connectionPoolMaxSize)
clusterConfig.setMasterConnectionMinimumIdleSize(connectionPoolMinSize)
clusterConfig.addNodeAddress(redisPrefix + instanceNodeAndPort())
clusterConfig.setTimeout(redisServerTimeout)
clusterConfig.setReadMode(ReadMode.SLAVE)
val client = Redisson.create(config)

// Adding data to Redis
val rMapCache = client.getMapCache("redis_store")
rMapCache.fastPutAsync(myRedisKey, myRedisData, ttl, timeUnit)
println(s"Added data for [key=$myRedisKey]")

// Printing Redis Slot for key (Only for testing)
val rKeys = client.getKeys
val slot = rKeys.getSlot(myRedisKey)
println(s"Slot partitioning [slot=$slot] [key=$myRedisKey]")

// Fetch data using getWithTTLOnly to read from replica nodes.
val fetchRedisData = rMapCache.getWithTTLOnlyAsync(myRedisKey)
println(s"Fetched data for [key=$redisKey]")

I have used this application code to put various keys into Redis in all the three stages on my AWS ElastiCache setup. While transitioning between each stage, I can see Redisson has identified the shard addition/removal and the ClusterConnectionManager is logging slot movements (... slots added to ..., ... slots removed from ...). I can also see that all the shards in all the stages have active connections with my application based on the “Current Connections” AWS graph of each shard.

Problem/Issue Faced: But in all 3 stages, some shards are never receiving any data at all. And I believe these shards should be receiving the data because my slot partitioning logline is printing the key’s slot to be on these no-data-receiving shards.

Is it possible that I am logging the slot partitioning incorrectly here, and Redisson is appending some metadata to $myRedisKey before inserting it into Redis? Or have I misconfigured something in this entire setup?

Please find attached some logs and graphs supporting that data for a key that should be going to shard#3 in stage#2 when there are 4 shards in total, but there are no items at all on shard#3.

// Masking key for security reasons
2021-10-24 00:26:00,171 -0700 INFO Service is my_redis_cluster.
2021-10-24 00:26:00,171 -0700 INFO Added data for [key= ******d80ae9zj****4ckq4goz****]
2021-10-24 00:26:00,172 -0700 INFO Slot partitioning [slot=6901] [key=******d80ae9zj****4ckq4goz****]
2021-10-24 00:26:00,172 -0700 INFO Fetched data for [key=******d80ae9zj****4ckq4goz****]

Shards Configuration

Shard#3 Current Item Count

Shard#3 Current Cache Hits

Happy to share more details.

[mrniko]

You can use RBucket object instead.

[calvincodes]

I just came across this wiki page describing that data partitioning for Redisson Java objects is only supported in Redisson Pro version. Is there a way to use Redisson as a (vanilla) Java client to store key-value-ttl entries in Redis, without the use of Redisson Java objects, that can support the use case that I have described above?