ioredis cluster doesn't discover newly added nodes

I’m finding that ioredis cluster doesn’t discover newly added nodes. On my application startup all nodes in the cluster will be discovered but if I add another node later on ioredis doesn’t see it.

Here are my steps to reproduce:

1. Start a set of redis instances with the redis create-cluster script: https://github.com/antirez/redis/tree/3.2.11/utils/create-cluster
2. Create a cluster using the same script
3. Start an ioredis client with this example script: https://gist.github.com/stephendeyoung/79910f6e2970ca163f9dd38d51776d6b. The script will seed Redis with 120000 keys that are distributed amongst the cluster. Then every five seconds it will count the number of keys across the whole cluster. Initially it counts the correct number of keys.
4. Start a new redis-server like this: redis-server --port 30007 --cluster-enabled yes --cluster-config-file nodes-30007.conf --cluster-node-timeout 2000 --appendonly yes --appendfilename appendonly-30007.aof --dbfilename dump-30007.rdb --logfile 30007.log --daemonize yes
5. Use the redis-trib.rb script to add the new node to the cluster: redis-trib.rb add-node 127.0.0.1:30007 127.0.0.1:30001
6. Reshard the cluster: redis-trib.rb reshard 127.0.0.1:30001
7. The script will continue to log the number of keys in the cluster but after resharding rather than counting 120000 keys it will count less than this

My expectation is that after adding the new node and resharding redis, ioredis will discover the new node and continue to count the correct number of keys.

Am I correct in expecting this behaviour? Please let me know if I’ve done anything wrong here.

[jcstanaway]

I’ve ended up doing that in my client app: periodically do CLUSTER SLOTS and compare the number of nodes with #nodes(). If #nodes < CLUSTER SLOTS nodes, then I do a cluster.disconnect(true) which appears to work. Though this doesn’t handle some odd scenario where a slave was moved from one master to another. And it has the downside of temporarily making redis unavailable to the app. It would be better if ioredis could handle this without causing the app to disconnect from the cluster.

What I noticed is if I killed a node, ioredis would shortly after completely remove that node from it’s list (#nodes().length would decrement). So even when the node recovers and rejoins the cluster, ioredis doesn’t reconnect to that node (unless a MOVED or ASK error is received which normally would not happen in a simple node failure / recovery scenario).

Chris

On Fri, Feb 16, 2018 at 8:23 PM, Zihua 子骅 [e-mail] wrote:

ioredis doesn’t fetch all the master list periodically from the servers, only when the MOVED or ASK error is received. That makes the result of #nodes() not up to date though.

We can refresh the master list in the background periodically to solve the problem. I’ll look into that.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_luin_ioredis_issues_585-23issuecomment-2D366408724&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=N1Msa0ZexCqrdtWLgqlbMZlNrFKF4YmyP5otZCNJ7mmERCt9HPx5t646YC5_jvy8&m=SDcwXnE76OG1Ewjg-JrcIoIkHggCrF1UimijVp6bthc&s=JKgkQtfivsppG7w6BnZTbdEAHdwFYB0pGwUypW9CguA&e=, or mute the thread https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AVxAa1oe30yAbTE79WUE-2Dz-2DQONs5qZdIks5tVjgmgaJpZM4SB4SF&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=N1Msa0ZexCqrdtWLgqlbMZlNrFKF4YmyP5otZCNJ7mmERCt9HPx5t646YC5_jvy8&m=SDcwXnE76OG1Ewjg-JrcIoIkHggCrF1UimijVp6bthc&s=HK3Eqr3GFzTDM8zkqeRlwOt6-Yy1BzWfzwllmgFpq3M&e= .

[menocomp]

I have that issue as well, in Sep 2022 😦