Out of Direct Memory because of IdleStateHandler

Expected behavior

When I use netty as a server, I use IdleStateHandler to keep heartbeat for each client. When a lot of clients connected the server should works fine.

Actual behavior

When a lot of clients connected, the server raised OutOfDirectMemoryError.

Steps to reproduce

1. Create instance of IdleStateHandler to make readeridle time bigger than 0. new IdleStateHandler(60, 0, 0, TimeUnit.SECONDS)
2. Connect a lot of clients to the server.

Minimal yet complete reproducer code (or URL to code)

Netty version

4.1.6 Final

JVM version (e.g. java -version)

java version “1.8.0_144” Java™ SE Runtime Environment (build 1.8.0_144-b01) Java HotSpot™ 64-Bit Server VM (build 25.144-b01, mixed mode)

OS version (e.g. uname -a)

Linux PW6J1216 3.0.101-63-default #1 SMP Tue Jun 23 16:02:31 UTC 2015 (4b89d0c) x86_64 x86_64 x86_64 GNU/Linux

###################################################### I found this issue is because of the state was not well protected in multi-thread enviroment. The code io.netty.handler.timeout.IdleStateHandler.java(from version 4.1.6 to netty-all-5.0.0.Alpha2): `

 private void initialize(ChannelHandlerContext ctx) {
    switch (state) {   // Mark A. Here can reach many times
    case 1:
    case 2:
        return;
    }

   state = 1;   // Mark B. Here can reach many times

    EventExecutor loop = ctx.executor();

    lastReadTime = lastWriteTime = System.nanoTime();
    if (readerIdleTimeNanos > 0) {
        readerIdleTimeout = loop.schedule(
                new ReaderIdleTimeoutTask(ctx),  // Mark C. Here can reach many times
                readerIdleTimeNanos, TimeUnit.NANOSECONDS);
    }
    if (writerIdleTimeNanos > 0) {
        writerIdleTimeout = loop.schedule(
                new WriterIdleTimeoutTask(ctx),  // Mark D. Here can reach many times
                writerIdleTimeNanos, TimeUnit.NANOSECONDS);
    }
    if (allIdleTimeNanos > 0) {
        allIdleTimeout = loop.schedule(
                new AllIdleTimeoutTask(ctx), // Mark F. Here can reach many times
                allIdleTimeNanos, TimeUnit.NANOSECONDS);
    }
}

`

When code executing in multi-thread ENV, many threads can reach the place of Mark A/Mark B at the same time(in both version of 4.1.6 and even the lastest version netty-all-5.0.0.Alpha2), so the place mark as Mark C/D/F can reach many times. I found code changed in netty-all-5.0.0.Alpha2, the member state changed from “private int state” to “private volatile int state”, but it still not multi-thread safe. I prefer to change it to “private AtomicInteger state = new AtomicInteger(0)”, here is my code:

` public class IdleStateHandler extends ChannelDuplexHandler { … private AtomicInteger state = new AtomicInteger(0); // 0 - none, 1 - initialized, 2 - destroyed … private void initialize(ChannelHandlerContext ctx) { if(state.compareAndSet(0, 1)) { EventExecutor loop = ctx.executor();

        lastReadTime = lastWriteTime = System.nanoTime();
        if (readerIdleTimeNanos > 0) {
            readerIdleTimeout = loop.schedule(
                    new ReaderIdleTimeoutTask(ctx),
                    readerIdleTimeNanos, TimeUnit.NANOSECONDS);
        }
        if (writerIdleTimeNanos > 0) {
            writerIdleTimeout = loop.schedule(
                    new WriterIdleTimeoutTask(ctx),
                    writerIdleTimeNanos, TimeUnit.NANOSECONDS);
        }
        if (allIdleTimeNanos > 0) {
            allIdleTimeout = loop.schedule(
                    new AllIdleTimeoutTask(ctx),
                    allIdleTimeNanos, TimeUnit.NANOSECONDS);
        }
    }
}

private void destroy() {
    state.set(2);

    if (readerIdleTimeout != null) {
        readerIdleTimeout.cancel(false);
        readerIdleTimeout = null;
    }
    if (writerIdleTimeout != null) {
        writerIdleTimeout.cancel(false);
        writerIdleTimeout = null;
    }
    if (allIdleTimeout != null) {
        allIdleTimeout.cancel(false);
        allIdleTimeout = null;
    }
}

…

}`

[amizurov]

@LiangKongRong HI, Do you use IdleStateHandler in shared mode or create new instance per channel ?

[normanmaurer]

@LiangKongRong I am happy to review a PR … Please provide one if you think you can improve it.