Redis replies immediately but Lettuce experiences random timeout failures

Bug Report

We have an application running on production which use lettuce to interact with redis, del operation mostly. Recently we found command timeout occurred randomly after startup 1-2 hours, we captured packets while client command timeout and found redis replay immediate(<1ms), but client get timeout.

• packet 431536: DEL request
• packet 431557: response 1
• packet 431559: ack for last packet

Current Behavior

command: del, args: [[key**********************************]], spend: 2000ms

Input Code

Input Code

// initialize client
ClientResources clientResources = DefaultClientResources.builder()
                .eventLoopGroupProvider(eventLoopGroupProvider) // reuse cross multi client
                .eventExecutorGroup(eventExecutorGroup) // same reused
                .build();
ClusterTopologyRefreshOptions refreshOptions = ClusterTopologyRefreshOptions.builder()
                .enableAdaptiveRefreshTrigger(
                        ClusterTopologyRefreshOptions.RefreshTrigger.MOVED_REDIRECT,
                        ClusterTopologyRefreshOptions.RefreshTrigger.ASK_REDIRECT)
                .build();
RedisClusterClient client = RedisClusterClient.create(resources, uri);
                client.setOptions(ClusterClientOptions.builder()
                        .socketOptions(buildSocketOptions()) // TCP_NODELAY, SO_KEEPALIVE
                        .timeoutOptions(buildTimeoutOptions()) // timeout 2s
                        .topologyRefreshOptions(refreshOptions)
                        .requestQueueSize(2000)
                        .build();
// call del operation
connection.sync().del(key);

Expected behavior/code

Environment

• Lettuce version(s): [5.1.2.RELEASE, 5.2.2.RELEASE]
• Netty versions: [4.1.29.Final, 4.1.48.Final]
• Redis version: [4.0.14]

Possible Solution

Additional context

[Shawyeok]

Note that custom codecs may lead to such a situation when encoding/decoding fails. Please make sure that this isn’t the case so that you log encoding exceptions from within your codec.

Problem resolved, you are right in the first place. @mp911de

We enabled TRACE logging and found io.netty.handler.codec.EncoderException in the log. We found a code issue that will cause a ConcurrentModificationException in the command encode process, therefor cause a codec encode error. We cannot catch this error normally because just call setAsync and didn’t wait for result at all. 😦

[mp911de]

Note that custom codecs may lead to such a situation when encoding/decoding fails. Please make sure that this isn’t the case so that you log encoding exceptions from within your codec.