Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Exceptions when changing node type in AWS Elasticache on the fly

See original GitHub issue

We tried to update underlying node type in our Elasticache Redis cluster. When AWS starts switching to newly created nodes, we see a lot of exception:

org.redisson.client.RedisException: MOVED redirection loop detected. Node redis://<node_address>:6379 has further redirect to redis://<same_node_address>:6379
	at org.redisson.command.RedisExecutor.checkAttemptPromise(RedisExecutor.java:404) ~[redisson-3.13.5.jar:3.13.5]
	at org.redisson.command.RedisExecutor.lambda$execute$3(RedisExecutor.java:164) ~[redisson-3.13.5.jar:3.13.5]
	at org.redisson.misc.RedissonPromise.lambda$onComplete$0(RedissonPromise.java:183) ~[redisson-3.13.5.jar:3.13.5]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:500) ~[netty-common-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:493) ~[netty-common-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:472) ~[netty-common-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:413) ~[netty-common-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:538) ~[netty-common-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:531) ~[netty-common-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:111) ~[netty-common-4.1.38.Final.jar:4.1.38.Final]
	at org.redisson.misc.RedissonPromise.tryFailure(RedissonPromise.java:96) ~[redisson-3.13.5.jar:3.13.5]
	at org.redisson.client.protocol.CommandData.tryFailure(CommandData.java:78) ~[redisson-3.13.5.jar:3.13.5]
	at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:339) ~[redisson-3.13.5.jar:3.13.5]
	at org.redisson.client.handler.CommandDecoder.decodeCommand(CommandDecoder.java:196) ~[redisson-3.13.5.jar:3.13.5]
	at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:134) ~[redisson-3.13.5.jar:3.13.5]
	at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:104) ~[redisson-3.13.5.jar:3.13.5]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) ~[netty-codec-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366) ~[netty-codec-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) ~[netty-codec-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[netty-transport-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[netty-transport-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[netty-transport-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) ~[netty-handler-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224) ~[netty-handler-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271) ~[netty-handler-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) ~[netty-codec-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) ~[netty-codec-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) ~[netty-codec-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[netty-transport-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[netty-transport-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[netty-transport-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) ~[netty-transport-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[netty-transport-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[netty-transport-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) ~[netty-transport-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[netty-transport-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697) ~[netty-transport-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632) ~[netty-transport-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549) ~[netty-transport-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511) ~[netty-transport-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) ~[netty-common-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.38.Final.jar:4.1.38.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.38.Final.jar:4.1.38.Final]
	at java.lang.Thread.run(Thread.java:829) [?:?]

Is it possible to scale Redis cluster vertically on the fly using Redisson?

Engine Version Compatibility: 6.0.5 Redisson version: 3.13.5, also tried 3.16.1

Issue Analytics

State:
Created 2 years ago
Comments:30 (29 by maintainers)

Top GitHub Comments

1reaction

mrnikocommented, Aug 30, 2021

Thanks for feedback!

Please try attached version. redisson-3.16.2-SNAPSHOT.jar.zip

1reaction

mrnikocommented, Aug 27, 2021

Fixed version attached.

redisson-3.16.2-SNAPSHOT.jar.zip

Top Results From Across the Web

Troubleshoot changing an ElastiCache Redis node type - AWS

The following are common reasons for problems changing your Redis node type: Insufficient memory on the target node type.

Amazon ElastiCache error messages - AWS Documentation

Solution: Change your request so that the total nodes in the region across all clusters for this account does not exceed %n. Or,...

Troubleshooting - Amazon ElastiCache for Redis

For such cases, it's a best practice to scale-up (change node type), scale-out (add shards in cluster-mode enabled clusters), reduce the number of...

ElastiCache for Redis components and features

ElastiCache supports changing a Redis (cluster mode disabled) cluster's node type to a larger node type dynamically. For information on scaling up or...

How to work with Cluster Mode on Amazon ElastiCache for ...

With the exception of the primary node, if a node fails for any reason, you do not lose data as it ... (vertical...