Threads blocked with IN_NATIVE state inside Netty

Expected behavior

All the threads managed by Netty need to co-run to complete all Network I/O initiated from multiple application threads that use Netty.

Actual behavior

We noticed that the system is unable to move further (i.e, JVM Hang) to process new resource creation requests via HTTP RESTful API, as all such calls ended up not having threads to be used to complete the request.

Across 3 thread-dumps spaced between 10-15 seconds , Out of 750 threads available on the system during a Jstack dump, atleast 1/3rd of the JVM threads have blocked on Netty (207 threads).

Thread 25375 - threadId:Thread 25375 - state:IN_NATIVE stackTrace:

• io.netty.channel.epoll.Native.epollWait0(int, long, int, int) @bci=0 (Compiled frame; information may be imprecise)
• io.netty.channel.epoll.Native.epollWait(int, io.netty.channel.epoll.EpollEventArray, int) @bci=10, line=109 (Compiled frame)
• io.netty.channel.epoll.EpollEventLoop.epollWait(boolean) @bci=117, line=232 (Compiled frame)
• io.netty.channel.epoll.EpollEventLoop.run() @bci=65, line=256 (Compiled frame)
• io.netty.util.concurrent.SingleThreadEventExecutor$2.run() @bci=13, line=112 (Interpreted frame)
• io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run() @bci=4, line=145 (Interpreted frame)
• java.lang.Thread.run() @bci=11, line=748 (Compiled frame)

Steps to reproduce

We have not come up yet with the shortest way to reproduce the problem.

Minimal yet complete reproducer code (or URL to code)

Netty version

3.10.6.Final

JVM version (e.g. java -version)

java version “1.8.0_141” Java™ SE Runtime Environment (build 1.8.0_141-b15) Java HotSpot™ 64-Bit Server VM (build 25.141-b15, mixed mode)

OS version (e.g. uname -a)

Linux cic-2.domain.tld 4.4.0-78-generic #99~14.04.2-Ubuntu SMP Thu Apr 27 18:49:46 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[normanmaurer]

Let me close this as I think its not an actual issue (if its an issue then its one because you or the library you are using is creating to many EventLoops). Please re-open if you still think there is an issue and provide more info why you think so.

[normanmaurer]

@vivek-konnect again I think there is nothing wrong with have threads sitting in this state… Its expected if there is no IO ready to be handled atm. The number of threads really depends on how many *EventLoopGroups you have but that’s up to you.