Why are slaves constantly added and removed in Redis AWS Elasticache
See original GitHub issueWe are running in a clustered configuration in elasticache and with the redisson client. We constantly get slaves being added and removed. Are there connectivity issues or other known issues using Elasticache?
2022-11-03 13:55:47
24 connections initialized for <ip3>/<ip3>:6379
2022-11-03 13:55:47
slave: rediss://<ip3>:6379 added for slot ranges: [[0-8191]]
2022-11-03 13:55:47
1 connections initialized for <ip3>/<ip3>:6379
2022-11-03 13:55:42
slave rediss://<ip3>:6379 removed for slot ranges: [[0-8191]]
2022-11-03 13:24:53
24 connections initialized for <ip3>/<ip3>:6379
2022-11-03 13:24:53
slave: rediss://<ip3>:6379 added for slot ranges: [[0-8191]]
2022-11-03 13:24:53
1 connections initialized for <ip3>/<ip3>:6379
2022-11-03 13:24:48
slave rediss://<ip3>:6379 removed for slot ranges: [[0-8191]]
2022-11-03 13:22:22
24 connections initialized for <ip6>/<ip6>:6379
2022-11-03 13:22:22
slave: rediss://<ip6>:6379 added for slot ranges: [[8192-16383]]
2022-11-03 13:22:22
1 connections initialized for <ip6>/<ip6>:6379
2022-11-03 13:22:17
slave rediss://<ip6>:6379 removed for slot ranges: [[8192-16383]]
2022-11-03 13:04:05
24 connections initialized for <ip3>/<ip3>:6379
2022-11-03 13:04:05
slave: rediss://<ip3>:6379 added for slot ranges: [[0-8191]]
2022-11-03 13:04:05
1 connections initialized for <ip3>/<ip3>:6379
2022-11-03 13:04:00
slave rediss://<ip3>:6379 removed for slot ranges: [[0-8191]]
2022-11-03 12:46:17
24 connections initialized for <ip6>/<ip6>:6379
2022-11-03 12:46:17
slave: rediss://<ip6>:6379 added for slot ranges: [[8192-16383]]
2022-11-03 12:46:17
1 connections initialized for <ip6>/<ip6>:6379
2022-11-03 12:46:12
slave rediss://<ip6>:6379 removed for slot ranges: [[8192-16383]]
2022-11-03 12:34:25
24 connections initialized for <ip4>/<ip4>:6379
2022-11-03 12:34:25
slave: rediss://<ip4>:6379 added for slot ranges: [[8192-16383]]
2022-11-03 12:34:25
1 connections initialized for <ip4>/<ip4>:6379
2022-11-03 12:34:20
slave rediss://<ip4>:6379 removed for slot ranges: [[8192-16383]]
2022-11-03 12:30:15
24 connections initialized for <ip5>/<ip5>:6379
2022-11-03 12:30:15
slave: rediss://<ip5>:6379 added for slot ranges: [[0-8191]]
2022-11-03 12:30:14
1 connections initialized for <ip5>/<ip5>:6379
2022-11-03 12:30:09
slave rediss://<ip5>:6379 removed for slot ranges: [[0-8191]]
2022-11-03 12:21:03
24 connections initialized for <ip5>/<ip5>:6379
2022-11-03 12:21:03
slave: rediss://<ip5>:6379 added for slot ranges: [[0-8191]]
2022-11-03 12:21:03
1 connections initialized for <ip5>/<ip5>:6379
2022-11-03 12:20:58
slave rediss://<ip5>:6379 removed for slot ranges: [[0-8191]]
2022-11-03 12:14:22
24 connections initialized for <ip6>/<ip6>:6379
2022-11-03 12:14:22
slave: rediss://<ip6>:6379 added for slot ranges: [[8192-16383]]
2022-11-03 12:14:22
1 connections initialized for <ip6>/<ip6>:6379
2022-11-03 12:14:17
slave rediss://<ip6>:6379 removed for slot ranges: [[8192-16383]]
2022-11-03 12:12:22
24 connections initialized for <ip3>/<ip3>:6379
2022-11-03 12:12:22
slave: rediss://<ip3>:6379 added for slot ranges: [[0-8191]]
2022-11-03 12:12:22
1 connections initialized for <ip3>/<ip3>:6379
2022-11-03 12:12:17
slave rediss://<ip3>:6379 removed for slot ranges: [[0-8191]]
2022-11-03 11:45:43
24 connections initialized for <ip4>/<ip4>:6379
2022-11-03 11:45:43
slave: rediss://<ip4>:6379 added for slot ranges: [[8192-16383]]
2022-11-03 11:45:43
1 connections initialized for <ip4>/<ip4>:6379
2022-11-03 11:45:38
slave rediss://<ip4>:6379 removed for slot ranges: [[8192-16383]]
2022-11-03 11:34:11
24 connections initialized for <ip3>/<ip3>:6379
2022-11-03 11:34:11
slave: rediss://<ip3>:6379 added for slot ranges: [[0-8191]]
2022-11-03 11:34:11
1 connections initialized for <ip3>/<ip3>:6379
2022-11-03 11:34:06
slave rediss://<ip3>:6379 removed for slot ranges: [[0-8191]]
2022-11-03 11:29:31
24 connections initialized for <ip6>/<ip6>:6379
2022-11-03 11:29:31
slave: rediss://<ip6>:6379 added for slot ranges: [[8192-16383]]
2022-11-03 11:29:30
1 connections initialized for <ip6>/<ip6>:6379
2022-11-03 11:29:25
slave rediss://<ip6>:6379 removed for slot ranges: [[8192-16383]]
2022-11-03 11:25:45
24 connections initialized for <ip6>/<ip6>:6379
2022-11-03 11:25:45
slave: rediss://<ip6>:6379 added for slot ranges: [[8192-16383]]
2022-11-03 11:25:45
1 connections initialized for <ip6>/<ip6>:6379
2022-11-03 11:25:40
slave rediss://<ip6>:6379 removed for slot ranges: [[8192-16383]]
2022-11-03 11:22:49
24 connections initialized for <ip5>/<ip5>:6379
2022-11-03 11:22:49
slave: rediss://<ip5>:6379 added for slot ranges: [[0-8191]]
2022-11-03 11:22:49
1 connections initialized for <ip5>/<ip5>:6379
2022-11-03 11:22:44
slave rediss://<ip5>:6379 removed for slot ranges: [[0-8191]]
2022-11-03 11:15:33
24 connections initialized for <ip6>/<ip6>:6379
2022-11-03 11:15:33
slave: rediss://<ip6>:6379 added for slot ranges: [[8192-16383]]
2022-11-03 11:15:33
1 connections initialized for <ip6>/<ip6>:6379
2022-11-03 11:15:28
slave rediss://<ip6>:6379 removed for slot ranges: [[8192-16383]]
2022-11-03 11:09:02
24 connections initialized for <ip4>/<ip4>:6379
2022-11-03 11:09:02
slave: rediss://<ip4>:6379 added for slot ranges: [[8192-16383]]
2022-11-03 11:09:02
1 connections initialized for <ip4>/<ip4>:6379
2022-11-03 11:08:57
slave rediss://<ip4>:6379 removed for slot ranges: [[8192-16383]]
2022-11-03 10:58:01
24 connections initialized for <ip4>/<ip4>:6379
2022-11-03 10:58:01
slave: rediss://<ip4>:6379 added for slot ranges: [[8192-16383]]
2022-11-03 10:58:00
1 connections initialized for <ip4>/<ip4>:6379
2022-11-03 10:57:55
slave rediss://<ip4>:6379 removed for slot ranges: [[8192-16383]]
2022-11-03 10:45:49
24 connections initialized for <ip3>/<ip3>:6379
2022-11-03 10:45:49
slave: rediss://<ip3>:6379 added for slot ranges: [[0-8191]]
2022-11-03 10:45:49
1 connections initialized for <ip3>/<ip3>:6379
2022-11-03 10:45:44
slave rediss://<ip3>:6379 removed for slot ranges: [[0-8191]]
2022-11-03 10:22:01
24 connections initialized for <ip6>/<ip6>:6379
2022-11-03 10:22:01
slave: rediss://<ip6>:6379 added for slot ranges: [[8192-16383]]
2022-11-03 10:22:00
1 connections initialized for <ip6>/<ip6>:6379
2022-11-03 10:21:55
slave rediss://<ip6>:6379 removed for slot ranges: [[8192-16383]]
2022-11-03 10:21:05
24 connections initialized for <ip6>/<ip6>:6379
2022-11-03 10:21:05
slave: rediss://<ip6>:6379 added for slot ranges: [[8192-16383]]
2022-11-03 10:21:05
1 connections initialized for <ip6>/<ip6>:6379
2022-11-03 10:21:00
slave rediss://<ip6>:6379 removed for slot ranges: [[8192-16383]]
2022-11-03 09:48:46
24 connections initialized for <ip4>/<ip4>:6379
2022-11-03 09:48:46
slave: rediss://<ip4>:6379 added for slot ranges: [[8192-16383]]
2022-11-03 09:48:46
1 connections initialized for <ip4>/<ip4>:6379
2022-11-03 09:48:41
slave rediss://<ip4>:6379 removed for slot ranges: [[8192-16383]]
2022-11-03 09:44:45
24 connections initialized for <ip5>/<ip5>:6379
2022-11-03 09:44:45
slave: rediss://<ip5>:6379 added for slot ranges: [[0-8191]]
2022-11-03 09:44:45
1 connections initialized for <ip5>/<ip5>:6379
2022-11-03 09:44:40
slave rediss://<ip5>:6379 removed for slot ranges: [[0-8191]]
2022-11-03 09:42:05
24 connections initialized for <ip3>/<ip3>:6379
2022-11-03 09:42:05
slave: rediss://<ip3>:6379 added for slot ranges: [[0-8191]]
2022-11-03 09:42:04
1 connections initialized for <ip3>/<ip3>:6379
2022-11-03 09:41:59
slave rediss://<ip3>:6379 removed for slot ranges: [[0-8191]]
2022-11-03 09:40:09
24 connections initialized for <ip4>/<ip4>:6379
2022-11-03 09:40:09
slave: rediss://<ip4>:6379 added for slot ranges: [[8192-16383]]
2022-11-03 09:40:09
1 connections initialized for <ip4>/<ip4>:6379
2022-11-03 09:40:04
slave rediss://<ip4>:6379 removed for slot ranges: [[8192-16383]]
2022-11-03 09:36:08
24 connections initialized for <ip4>/<ip4>:6379
2022-11-03 09:36:08
slave: rediss://<ip4>:6379 added for slot ranges: [[8192-16383]]
2022-11-03 09:36:08
1 connections initialized for <ip4>/<ip4>:6379
2022-11-03 09:36:03
slave rediss://<ip4>:6379 removed for slot ranges: [[8192-16383]]
2022-11-03 09:29:02
24 connections initialized for <ip3>/<ip3>:6379
2022-11-03 09:29:02
slave: rediss://<ip3>:6379 added for slot ranges: [[0-8191]]
2022-11-03 09:29:02
1 connections initialized for <ip3>/<ip3>:6379
2022-11-03 09:28:57
slave rediss://<ip3>:6379 removed for slot ranges: [[0-8191]]
2022-11-03 09:22:26
24 connections initialized for <ip3>/<ip3>:6379
2022-11-03 09:22:26
slave: rediss://<ip3>:6379 added for slot ranges: [[0-8191]]
2022-11-03 09:22:26
1 connections initialized for <ip3>/<ip3>:6379
2022-11-03 09:22:21
slave rediss://<ip3>:6379 removed for slot ranges: [[0-8191]]
2022-11-03 09:14:44
24 connections initialized for <ip4>/<ip4>:6379
2022-11-03 09:14:44
master: rediss://<ip2>:6379 added for slot ranges: [[8192-16383]]
2022-11-03 09:14:44
slaves: [rediss://<ip6>:6379, rediss://<ip4>:6379] added for slot ranges: [[8192-16383]]
2022-11-03 09:14:44
24 connections initialized for <ip6>/<ip6>:6379
2022-11-03 09:14:44
24 connections initialized for <ip3>/<ip3>:6379
2022-11-03 09:14:44
master: rediss://<ip1>:6379 added for slot ranges: [[0-8191]]
2022-11-03 09:14:44
slaves: [rediss://<ip3>:6379, rediss://<ip5>:6379] added for slot ranges: [[0-8191]]
2022-11-03 09:14:44
24 connections initialized for <ip5>/<ip5>:6379
2022-11-03 09:14:44
1 connections initialized for <ip6>/<ip6>:6379
2022-11-03 09:14:44
1 connections initialized for <ip4>/<ip4>:6379
2022-11-03 09:14:44
24 connections initialized for <ip2>/<ip2>:6379
2022-11-03 09:14:44
1 connections initialized for <ip5>/<ip5>:6379
2022-11-03 09:14:44
1 connections initialized for <ip3>/<ip3>:6379
2022-11-03 09:14:44
24 connections initialized for <ip1>/<ip1>:6379
2022-11-03 09:14:44
1 connections initialized for <ip2>/<ip2>:6379
2022-11-03 09:14:44
1 connections initialized for <ip1>/<ip1>:6379
2022-11-03 09:14:43
Redis cluster nodes configuration got from clustercfg.<redacted>.amazonaws.com/<ip1>:6379:
1ba5b00658cec7a94e4c7f29b8985c7babb4fcff <redacted>-0001-002.<redacted>cache.amazonaws.com:6379@1122 slave 6d476cddb978d42efdd93969cbb96eef4c71d3d9 0 1667488481483 2 connected
19c394a61f328d874d6752cdacb8111704717efa <redacted>-0002-001.<redacted>cache.amazonaws.com:6379@1122 master - 0 1667488483522 1 connected 8192-16383
db9dfe39bad1aad9a9c51d5b1b7ace94e3bb7309 <redacted>-0001-003.<redacted>cache.amazonaws.com:6379@1122 slave 6d476cddb978d42efdd93969cbb96eef4c71d3d9 0 1667488482503 2 connected
8c51e839d5ecae880359086a96d53c23523ea26d <redacted>-0002-002.<redacted>cache.amazonaws.com:6379@1122 slave 19c394a61f328d874d6752cdacb8111704717efa 0 1667488480000 1 connected
d38aced77b8dcb97159d9da169dcc30261169b51 <redacted>-0002-003.<redacted>cache.amazonaws.com:6379@1122 slave 19c394a61f328d874d6752cdacb8111704717efa 0 1667488482000 1 connected
6d476cddb978d42efdd93969cbb96eef4c71d3d9 <redacted>-0001-001.<redacted>cache.amazonaws.com:6379@1122 myself,master - 0 1667488481000 2 connected 0-8191
Show context
2022-11-03 09:14:43
Redisson 3.17.7
2022-11-03 09:14:43
createRedissonClient: configuring clustering with nodeAddresses=[rediss://clustercfg.<redacted>cache.amazonaws.com:6379]
Issue Analytics
- State:
- Created a year ago
- Comments:12 (3 by maintainers)
Top Results From Across the Web
Redis-specific parameters - Amazon ElastiCache for Redis
The close-on-slave-write parameter is introduced by Amazon ElastiCache to give you more control over how your cluster responds when a primary node and...
Read more >ElastiCache for Redis Documentation history
ElastiCache for Redis has added support for adding and removing read replicas from any cluster with no cluster downtime. For more information about...
Read more >Learn why Redis client read requests are read from or ...
I'm using Amazon ElastiCache for Redis. Why are my Redis client read requests always read from or redirected to the primary node of...
Read more >Class: Aws::ElastiCache::Client — AWS SDK for Ruby V2
A Redis (cluster mode disabled) replication group is a collection of clusters, where one of the clusters is a read/write primary and the...
Read more >Supported ElastiCache for Redis versions - AWS Documentation
ElastiCache for Redis 7.0 adds a number of improvements and support for new ... for Redis 3.2.10 introduces online cluster resizing to add...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found

Please try the attached version and share log output.
redisson-3.18.1-SNAPSHOT.jar.zip
OK, I’ve added logging to a Dns resolver with the following results. The cluster nodes command is returning consistently and the DNS names are consistently resolving to the same ips. Also, what I see consistently is that the DNS resolver doesn’t seem to have finished with all of its calls (e.g. the call for the clustercfg.development-k8s-rg nodes) prior to a slave being removed.