Redis cluster refresh of large clusters keeps I/O threads busy

Bug Report

Current Behavior

For large redis cluster, using enablePeriodicRefresh causes serious performance problems.

The processing time required for DefaultClusterTopologyRefresh.getNodeSpecificViews increases in proportion to the cluster size. Unfortunately, this is executed in the NIO event loop thread.

See the repository README below for details. https://github.com/be-hase/lettuce-with-large-cluster

Input Code

I have prepared the code that can reproduce this problem. https://github.com/be-hase/lettuce-with-large-cluster

Expected behavior/code

Even if we use enablePeriodicRefresh with large redis cluster, the performance will not deteriorate.

Environment

• Lettuce version(s): 6.1.6.RELEASE
• Redis version: 6.2.0

Possible Solution

Idea 1

Stop running DefaultClusterTopologyRefresh.getNodeSpecificViews on the NIO event loop.

I won’t go into details, but if I customized DefaultClusterTopologyRefresh and ran it on another thread, the performance improved.

My team is considering adopting this method as a workaround.

Idea 2

Looking at the previous framegraph, it seems that the overhead is large in the processing using BitSet. Why not change to a more primitive method like boolean[] ?

However, I’m not sure if the performance will improve.

Additional context

NONE

[be-hase]

@mp911de Submited PR. Please check 🙏 https://github.com/lettuce-io/lettuce-core/pull/2048

[mp911de]

Feel free to submit a pull request to use ClientResources.eventExecutorGroup() in the future chaining in DefaultClusterTopologyRefresh.getNodeSpecificViews