Lots of warnings: Selector.select() returned prematurely 512 times in a row; rebuilding Selector io.netty.channel.nio.SelectedSelectionKeySetSelector@6edf74c3 after network issue.

Expected behavior

no issues

Actual behavior

We use Netty for message exchange between two internal systems. The system is running smoothly for months. After a network incident (inbound / outbound communication was affected) we’ve suddenly seen a lot of warning messages in our logs. We are running 4 machines, all are equal to each other. Nearly a the same time all machines got the same behavior.

Each system was the message logging nearly 60k times per minute Selector.select() returned prematurely 512 times in a row; rebuilding Selector io.netty.channel.nio.SelectedSelectionKeySetSelector@6edf74c3

When we restarted the Event loops (but not the JVM) the problem went away. The thread that is logging the warnings is the one that is responsible for sending out the response to the remote peer

This is, what we are actually doing:

receivingGroup = new NioEventLoopGroup(protocolServerConfiguration.getReceivingThreadSize(), receivingExecutors);
        respondingGroup = new NioEventLoopGroup(protocolServerConfiguration.getRespondingThreadSize(), respondingExecutors);
        serverBootstrap = new ServerBootstrap();
        serverBootstrap.group(receivingGroup, respondingGroup);
        serverBootstrap.channel(NioServerSocketChannel.class);
        serverBootstrap.childHandler(new ChunkedChannelInitializer());
...
protected void initChannel(SocketChannel socketChannel) {
    String remoteHostname = socketChannel.remoteAddress().getHostName();
    LOGGER.debug("New client connection from: {}.", remoteHostname);
    // read and parse the message from the remote peer
    ChannelInboundHandlerAdapter channelRequestInboundHandlerAdapter = new ChannelRequestInboundHandlerAdapter();
    socketChannel.pipeline().addLast(channelRequestInboundHandlerAdapter);            
}

// after the message is successfully parsed, we send the response
...
public ChannelFuture sendResponse(byte[] responseMessageBytes) {
    ByteBuf byteBuf = ByteBufAllocator.DEFAULT.heapBuffer(responseMessageBytes.length);
    byteBuf.writeBytes(responseMessageBytes);
    socketChannel.writeAndFlush(byteBuf);
    return socketChannel.close();
}

When the issue occurred the CPU usage went up to 100 percent on all machines.

Maybe this is related to #8698.

Steps to reproduce

We could not reproduce it. The system run for weeks without issues. Then the issue suddenly occurred. After restart it was gone again.

Minimal yet complete reproducer code (or URL to code)

Netty version

4.1.42.Final

JVM version (e.g. java -version)

OpenJDK 64-Bit Server VM (Zulu 8.48.0.52-SA-linux64) (build 25.262-b19, mixed mode)

OS version (e.g. uname -a)

Linux 3.10.0-1160.11.1.el7.x86_64 #1 SMP Mon Nov 30 13:05:31 EST 2020 x86_64 x86_64 x86_64 GNU/Linux

[PabloFiori]

For using Netty on Android where epoll doesn’t seem to work, is there any way to work around this @normanmaurer? I currently repro this with 100% consistency right now. I also double checked to make sure that there aren’t any memory leaks in our code. I unfortunately cannot share a reproducer.

[mnm364]

@PabloFiori this seems concerning, especially considering that Netty 4.1 claims to support Android >4.0 but this seems like a non-starter to use Netty to manage NIO-selectors. I wonder if there are grounds here to file an issue branching off of this discussion; this is also something I have run into.

[UPDATE] In fact, it seems this issue has come up already but was rejected as a change in https://github.com/netty/netty/pull/10881. 😕