Deadlock when starting server

Expected behavior

Server starts

Actual behavior

Sometimes server hangs

Steps to reproduce

This happens in the Infinispan test suite that starts and stops server dozens of times. There isn’t a reproducer at the moment

Minimal yet complete reproducer code (or URL to code)

Netty version

4.1.5.Final

JVM version (e.g. java -version)

openjdk version “1.8.0_121”

OS version (e.g. uname -a)

Linux gfernandes 4.9.10-100.fc24.x86_64 #1 SMP Wed Feb 15 18:35:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Details about the issue:

We observed a WARN during one of our netty based servers startup:

12:13:08,259 WARNING [io.netty.channel.epoll.EpollEventLoop] (MemcachedServerMaster-1-1) Unexpected exception in the selector loop.: java.lang.NullPointerException
	at io.netty.util.internal.shaded.org.jctools.queues.MpscChunkedArrayQueue.poll(MpscChunkedArrayQueue.java:264)
	at io.netty.util.concurrent.SingleThreadEventExecutor.pollTaskFrom(SingleThreadEventExecutor.java:223)
	at io.netty.util.concurrent.SingleThreadEventExecutor.pollTask(SingleThreadEventExecutor.java:218)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:306)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
	at java.lang.Thread.run(Thread.java:745)

Trimmed thread dump:

2017-02-23 13:43:18
Full thread dump OpenJDK 64-Bit Server VM (25.121-b14 mixed mode):

"Attach Listener" #184 daemon prio=9 os_prio=0 tid=0x00007f5fcc003800 nid=0x3749 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"MemcachedServerMaster-1-1" #123 prio=5 os_prio=0 tid=0x00007f5f9c7ba800 nid=0x2dfd runnable [0x00007f5f102e3000]
   java.lang.Thread.State: RUNNABLE
	at io.netty.util.internal.shaded.org.jctools.queues.MpscChunkedArrayQueue.poll(MpscChunkedArrayQueue.java:252)
	at io.netty.util.concurrent.SingleThreadEventExecutor.pollTaskFrom(SingleThreadEventExecutor.java:223)
	at io.netty.util.concurrent.SingleThreadEventExecutor.pollTask(SingleThreadEventExecutor.java:218)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:306)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
	at java.lang.Thread.run(Thread.java:745)


"MSC service thread 1-4" #17 prio=5 os_prio=0 tid=0x00007f5fa4001800 nid=0x2d85 in Object.wait() [0x00007f5fe5fb7000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:239)
	- locked <0x00000000f82f4920> (a io.netty.bootstrap.AbstractBootstrap$PendingRegistrationPromise)
	at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129)
	at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28)
	at io.netty.util.concurrent.DefaultPromise.sync(DefaultPromise.java:340)
	at io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:117)
	at io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:28)
	at org.infinispan.server.core.transport.NettyTransport.start(NettyTransport.java:136)
	at org.infinispan.server.core.AbstractProtocolServer.startTransport(AbstractProtocolServer.java:95)
	at org.infinispan.server.core.AbstractProtocolServer.startInternal(AbstractProtocolServer.java:73)
	at org.infinispan.server.memcached.MemcachedServer.startInternal(MemcachedServer.java:52)
	at org.infinispan.server.memcached.MemcachedServer.startInternal(MemcachedServer.java:30)
	at org.infinispan.server.core.AbstractProtocolServer.start(AbstractProtocolServer.java:80)
	at org.infinispan.server.endpoint.subsystem.SecurityActions$6.run(SecurityActions.java:136)
	at org.infinispan.server.endpoint.subsystem.SecurityActions$6.run(SecurityActions.java:133)
	at org.infinispan.security.Security.doPrivileged(Security.java:76)
	at org.infinispan.server.endpoint.subsystem.SecurityActions.doPrivileged(SecurityActions.java:42)
	at org.infinispan.server.endpoint.subsystem.SecurityActions.startProtocolServer(SecurityActions.java:140)
	at org.infinispan.server.endpoint.subsystem.ProtocolServerService.startProtocolServer(ProtocolServerService.java:190)
	at org.infinispan.server.endpoint.subsystem.ProtocolServerService.start(ProtocolServerService.java:148)
	- locked <0x00000000e1b23f88> (a org.infinispan.server.endpoint.subsystem.ProtocolServerService)
	at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

At this point the process is totally hang.

[nitsanw]

@johnou yep… at this point I don’t see a way to make sense of the error. There was an MPSC related issue fixed in the 2.0 release which may be related though: https://github.com/JCTools/JCTools/issues/135 So worth updating and seeing if it helps.

[johnou]

@gustavonalle you’ll want to pull in Netty 4.1.9 when released for the MPSC issue @nitsanw mentioned above, see https://github.com/netty/netty/pull/6399