Epoll client channel throws NotYetConnectedException on connect failure

I am observing a intermittent failure case where a client using the EpollSocketChannel transport will have a NotYetConnectedException raised on its pipeline as part of a connection attempt failing.

Expected behavior

Connection failures will result in the connect() future failing and not invoke any handlers in the channel pipeline.

Actual behavior

NotYetConnectedException is sometimes thrown on the pipeline for the failed channel:

(epollEventLoopGroup-3-16) {ChannelInboundHandler#exceptionCaught}: java.nio.channels.NotYetConnectedException: null
     at io.netty.channel.unix.Errors.ioResult(Errors.java:171)
     at io.netty.channel.unix.FileDescriptor.readAddress(FileDescriptor.java:143)
     at io.netty.channel.epoll.AbstractEpollChannel.doReadBytes(AbstractEpollChannel.java:348)
     at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:778)
     at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
     at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
     at java.lang.Thread.run(Thread.java:834)

Accompanied by ClosedChannelException for the actual connect future:

java.nio.channels.ClosedChannelException: null
     at io.netty.channel.AbstractChannel$AbstractUnsafe.newClosedChannelException(AbstractChannel.java:957)
     at io.netty.channel.AbstractChannel$AbstractUnsafe.ensureOpen(AbstractChannel.java:976)
     at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:552)
     at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342)
     at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:548)
     at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:533)
     at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495)
     at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51)
     at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296)
     at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:548)
     at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:533)
     at io.netty.handler.logging.LoggingHandler.connect(LoggingHandler.java:231)
     at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:548)
     at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:533)
     at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:517)
     at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978)
     at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:253)
     at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:250)
     at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
     at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
     at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
     at java.lang.Thread.run(Thread.java:834)

Steps to reproduce

Issue happens very intermittently and is difficult to reproduce on-demand.

I am not deeply familiar w/ Epoll so I am raising this issue to see if someone with more experience here might be able to spot some type of defect. I think these lines from the stack trace are relevant:

     at io.netty.channel.epoll.AbstractEpollChannel.doReadBytes(AbstractEpollChannel.java:348)
     at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:778)
     at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)

https://github.com/netty/netty/blob/4e86f768b916a34d3a929251eff8428ef47be65d/transport-native-epoll/src/main/java/io/netty/channel/epoll/EpollEventLoop.java#L468-L476

https://github.com/netty/netty/blob/4e86f768b916a34d3a929251eff8428ef47be65d/transport-native-epoll/src/main/java/io/netty/channel/epoll/AbstractEpollStreamChannel.java#L778

https://github.com/netty/netty/blob/4e86f768b916a34d3a929251eff8428ef47be65d/transport-native-epoll/src/main/java/io/netty/channel/epoll/AbstractEpollChannel.java#L347-L348

I noticed that commit https://github.com/netty/netty/commit/c7cb104dc48f179b61ed1146b6c0529c2e1115bc#diff-db3e069239a403b954e3ebc024ba9507L336 removed a && ch.isOpen() check that used to previously guard the epollInReady() call:

https://github.com/netty/netty/blob/dbbdbe11a6237cc08a9a6eafeeac9c7d80924305/transport-native-epoll/src/main/java/io/netty/channel/epoll/EpollEventLoop.java#L336

I’m wondering if that behavior is correct. Rather than dropping ch.isOpen() to support half-closed, should there at least be a check to see if the channel was ever open? I suspect this change might make sense from a server perspective but perhaps not a client perspective?

Netty version

4.1.50

[normanmaurer]

On my to do list for tomorrow…

[rishika728]

@normanmaurer, any ideas w.r.t the last few updates.