Handling slow clients / detecting idleness

Expected behavior

When using IdleStateHandler with observeOutput=true the IdleStateEvent.WRITER_IDLE or IdleStateEvent.ALL_IDLE should not trigger because of slow clients. This is same issue as in #6150.

Actual behavior

The IdleStateHandler triggers IdleStateEvent.WRITER_IDLE or IdleStateEvent.ALL_IDLE even though Netty is still writing to the client socket.

Steps to reproduce

Create an minimal server, use IdleStateHandler to detect idleness. Create an client that slowly downloads the content from the server. Watch the IdleStateHandler trigger before the download is complete.

Minimal yet complete reproducer code (or URL to code)

https://gist.github.com/magnus-gustafsson/c009c04aedf14a8b426dcb48450cc7d4 When this sample is run with the IdleStateHandler in the pipeline we get the following output:

Mon Mar 04 09:02:28 CET 2019: Server - listening on: http://localhost:8082/
Mon Mar 04 09:02:28 CET 2019: Client - connecting
Mon Mar 04 09:02:28 CET 2019: Server - Received request, writing response with size 67108864 on : [id: 0xb4450a55, L:/127.0.0.1:8082 - R:/127.0.0.1:38182]
Mon Mar 04 09:02:38 CET 2019: Server - Channel is idle, closing it: [id: 0xb4450a55, L:/127.0.0.1:8082 - R:/127.0.0.1:38182], ALL_IDLE
Mon Mar 04 09:02:38 CET 2019: Server - : Write complete. Success was false on [id: 0xb4450a55, L:/127.0.0.1:8082 ! R:/127.0.0.1:38182]
Mon Mar 04 09:02:41 CET 2019: Client - Done : read 20998517

There we see that the ALL_IDLE event is triggered. This results in that the client only manage to download 20998517 bytes.

However, if we remove IdleStateHandler from the pipeline we get:

Mon Mar 04 09:05:05 CET 2019: Server - listening on: http://localhost:8082/
Mon Mar 04 09:05:05 CET 2019: Client - connecting
Mon Mar 04 09:05:05 CET 2019: Server - Received request, writing response with size 67108864 on : [id: 0xf8335944, L:/127.0.0.1:8082 - R:/127.0.0.1:38252]
Mon Mar 04 09:05:44 CET 2019: Server - : Write complete. Success was true on [id: 0xf8335944, L:/127.0.0.1:8082 ! R:/127.0.0.1:38252]
Mon Mar 04 09:05:47 CET 2019: Client - Done : read 67108949

The client manage to download everything from the server.

Netty version

Same behavior in both 4.1.7.Final and 4.1.33.Final

JVM version (e.g. java -version)

java version “1.8.0_161” Java™ SE Runtime Environment (build 1.8.0_161-b12) Java HotSpot™ 64-Bit Server VM (build 25.161-b12, mixed mode)

OS version (e.g. uname -a)

Linux magnusg 4.9.0-040900-generic #201612111631 SMP Sun Dec 11 21:33:00 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[qeesung]

When using IdleStateHandler with observeOutput=true, IdleStateHandler detects the output data changes by executing method hasOutputChanged

private boolean hasOutputChanged(ChannelHandlerContext ctx, boolean first) {
    if (observeOutput) {
        Channel channel = ctx.channel();
        Unsafe unsafe = channel.unsafe();
        ChannelOutboundBuffer buf = unsafe.outboundBuffer();

        if (buf != null) {
            int messageHashCode = System.identityHashCode(buf.current());
            // pending write bytes will remain unchanged when flushing a large byte buffer entry.
            long pendingWriteBytes = buf.totalPendingWriteBytes();

            if (messageHashCode != lastMessageHashCode || pendingWriteBytes != lastPendingWriteBytes) {
                lastMessageHashCode = messageHashCode;
                lastPendingWriteBytes = pendingWriteBytes;

                if (!first) {
                    return true;
                }
            }
        }
    }
    return false;
}

I noticed that ChannelOutboundBuffer total pending write bytes size will remain unchanged when flushing a large byte buffer entry. It doesn’t change until all buffer data in entry is flushed into channel, so when IdleStateChannel encounters a slow client, hasOutputChanged returns false, triggering the IDLE event.

I just submitted a related issue #9005

[magnus-gustafsson]

Hi, I just rerun the test I attached to this issue with netty-all:4.1.35.Final and this bug has not been fixed in #9020

If the server writes a response that has an ByteBuf that is larger then what the client can download in the IdleStateHandler idle timeout, it will trigger the timeout.