Growing number of NioEventLoopGroup threads

I know that the number of threads in the worker group does not exceed the specified number(20). But being created new threads for new connection handling, it does not seem to be removed when the connection is closed. And it occurs OOM cause thread count continues to increase.

Expected behavior

Does not exceed the specified number(20).

Actual behavior

JMX results are confusing. thread count continues to increase. Below is the result of dumping the thread.

"nioEventLoopGroup-281-1" - Thread t@552
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <339ad506> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <58219b3f> (a java.util.Collections$UnmodifiableSet)
- locked <12c3e9e9> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:752)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:408)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers:
- None

Steps to reproduce

Here is my client code.

val (clientBossGroup, clientWorkerGroup) = (new NioEventLoopGroup(1), new NioEventLoopGroup(20))
try {
	val clientBootstrap = new ServerBootstrap()
	clientBootstrap.group(clientBossGroup, clientWorkerGroup)
		.channel(classOf[NioServerSocketChannel])
		.childHandler(new ChannelInitializer[SocketChannel]() {
			override def initChannel(ch: SocketChannel): Unit = {
				val pipeline = ch.pipeline()
				requires.sslContextOpt foreach { sslContext =>
					val sslEngine: SSLEngine = sslContext.createSSLEngine()
					sslEngine.setUseClientMode(false)
					pipeline.addLast(new SslHandler(sslEngine))
				}
				pipeline.addLast(new HttpServerCodec())
				pipeline.addLast(new HttpObjectAggregator(65536))
				pipeline.addLast(new HttpRequestHandler(clientRequestProcessor, requires.broadcastSuite, requires.nettySlackService)(requires.ecNetty))
			}
		})
		.option[java.lang.Integer](ChannelOption.SO_BACKLOG, 128)
		.childOption[java.lang.Boolean](ChannelOption.SO_KEEPALIVE, true)

	val clientFuture = clientBootstrap.bind(requires.configure.clientPortNumber).sync()
	println(s"(Client) Listening to port ${requires.configure.clientPortNumber}...")
	clientFuture.channel().closeFuture().sync()
} finally {
	clientBossGroup.shutdownGracefully()
	clientWorkerGroup.shutdownGracefully()
}

Minimal yet complete reproducer code (or URL to code)

Netty version

4.1.12.Final

JVM version (e.g. java -version)

openjdk version "1.8.0_162"
OpenJDK Runtime Environment (build 1.8.0_162-8u162-b12-1~deb9u1-b12)
OpenJDK 64-Bit Server VM (build 25.162-b12, mixed mode)

OS version (e.g. uname -a)

Linux 900f86cfe42d 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 GNU/Linux

[slandelle]

@sungjk Your code is weird. What you call “client” uses server-side code (ServerBootstrap, NioServerSocketChannel). The most likely reason for ending up with tons of threads is that you’re creating tons of Bootstraps (which is wrong) and not shutting them down (which is wrong too), hence @johnou asking you for an actionable reproducer.

[normanmaurer]

@sungjk I can not tell you how and why but you create 48 NioEventLoopGroup instances and each of these hold 8 EventLoops. This explains why you see so many threads…

So as you see this is an user error and not nettys fault.