Memory leak in latest netty version.
See original GitHub issueAfter recent update to 4.1.7-Final (from 4.1.4-Final) my servers started dying with OOM within few hours. Before they were running for weeks with no issues.
Error :
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 64 byte(s) of direct memory (used: 468189141, max: 468189184)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:614) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:568) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:68) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:625) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:65) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:170) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:131) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:73) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.channel.RecvByteBufAllocator$DelegatingHandle.allocate(RecvByteBufAllocator.java:124) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:956) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:359) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.safeExecute(SingleThreadEventExecutor.java:451) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:306) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877) ~[server-0.22.0-SNAPSHOT.jar:?]
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) ~[server-0.22.0-SNAPSHOT.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Or :
08:28:00.752 WARN - Failed to mark a promise as failure because it has succeeded already: DefaultChannelPromise@7cd20
32d(success)io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 18713 byte(s) of direct memory (used: 468184872, max: 468189184)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:631) ~[server-0.21.7-2.jar:?]
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:585) ~[server-0.21.7-2.jar:?]
at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30) ~[server-0.21.7-2.jar:?]
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:68) ~[server-0.21.7-2.jar:?]
at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25) ~[server-0.21.7-2.jar:?]
at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:624) ~[server-0.21.7-2.jar:?]
at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:65) ~[server-0.21.7-2.jar:?]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[server-0.21.7-2.jar:?]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:170) ~[server-0.21.7-2.jar:?]
at io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1533) ~[server-0.21.7-2.jar:?]
at io.netty.handler.ssl.SslHandler.allocateOutNetBuf(SslHandler.java:1544) ~[server-0.21.7-2.jar:?]
at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:575) ~[server-0.21.7-2.jar:?]
at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:550) ~[server-0.21.7-2.jar:?]
at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:531) ~[server-0.21.7-2.jar:?]
at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:1324) ~[server-0.21.7-2.jar:?]
at io.netty.handler.ssl.SslHandler.closeOutboundAndChannel(SslHandler.java:1307) ~[server-0.21.7-2.jar:?]
at io.netty.handler.ssl.SslHandler.close(SslHandler.java:498) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:625) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:609) ~[server-0.21.7-2.jar:?]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.close(CombinedChannelDuplexHandler.java:504) ~[server-0.21.7-2.jar:?]
at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71) ~[server-0.21.7-2.jar:?]
at io.netty.channel.CombinedChannelDuplexHandler.close(CombinedChannelDuplexHandler.java:315) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:625) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:609) ~[server-0.21.7-2.jar:?]
at io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:625) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:609) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:466) ~[server-0.21.7-2.jar:?]
at cc.blynk.server.core.protocol.handlers.DefaultExceptionHandler.handleUnexpectedException(DefaultExceptionHandler.java:59) ~[server-0.21.7-2.jar:?]
at cc.blynk.server.core.protocol.handlers.DefaultExceptionHandler.handleGeneralException(DefaultExceptionHandler.java:43) ~[server-0.21.7-2.jar:?]
at cc.blynk.core.http.handlers.StaticFileHandler.exceptionCaught(StaticFileHandler.java:277) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedC:
08:28:00.752 WARN - Failed to mark a promise as failure because it has succeeded already: DefaultChannelPromise@7cd20
32d(success)io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 18713 byte(s) of direct memory (used: 468184872, max: 468189184)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:631) ~[server-0.21.7-2.jar:?]
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:585) ~[server-0.21.7-2.jar:?]
at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30) ~[server-0.21.7-2.jar:?]
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:68) ~[server-0.21.7-2.jar:?]
at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25) ~[server-0.21.7-2.jar:?]
at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:624) ~[server-0.21.7-2.jar:?]
at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:65) ~[server-0.21.7-2.jar:?]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[server-0.21.7-2.jar:?]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:170) ~[server-0.21.7-2.jar:?]
at io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1533) ~[server-0.21.7-2.jar:?]
at io.netty.handler.ssl.SslHandler.allocateOutNetBuf(SslHandler.java:1544) ~[server-0.21.7-2.jar:?]
at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:575) ~[server-0.21.7-2.jar:?]
at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:550) ~[server-0.21.7-2.jar:?]
at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:531) ~[server-0.21.7-2.jar:?]
at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:1324) ~[server-0.21.7-2.jar:?]
at io.netty.handler.ssl.SslHandler.closeOutboundAndChannel(SslHandler.java:1307) ~[server-0.21.7-2.jar:?]
at io.netty.handler.ssl.SslHandler.close(SslHandler.java:498) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:625) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:609) ~[server-0.21.7-2.jar:?]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.close(CombinedChannelDuplexHandler.java:504) ~[server-0.21.7-2.jar:?]
at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71) ~[server-0.21.7-2.jar:?]
at io.netty.channel.CombinedChannelDuplexHandler.close(CombinedChannelDuplexHandler.java:315) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:625) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:609) ~[server-0.21.7-2.jar:?]
at io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:625) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:609) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:466) ~[server-0.21.7-2.jar:?]
at cc.blynk.server.core.protocol.handlers.DefaultExceptionHandler.handleUnexpectedException(DefaultExceptionHandler.java:59) ~[server-0.21.7-2.jar:?]
at cc.blynk.server.core.protocol.handlers.DefaultExceptionHandler.handleGeneralException(DefaultExceptionHandler.java:43) ~[server-0.21.7-2.jar:?]
at cc.blynk.core.http.handlers.StaticFileHandler.exceptionCaught(StaticFileHandler.java:277) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedC:
I did restart and made heap dump before abnormal memory consumption and after first error messages from above :
This screenshot shows difference between heap after server start (takes 17% of RAM of Instance) and first OOM in logs (takes 31% of RAM of instance). Instance RAM is 2 GB. So look like all direct memory was consumed (468MB) while heap itself takes less than direct buffers. Load on server is pretty low - 900 req/sec, with ~600 active connections. CPU consumption is only ~15%.
I tried to analyze heap dump but I don’t know netty well in order to make any conclusions.
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
<netty.version>4.1.7.Final</netty.version>
<netty.tcnative.version>1.1.33.Fork25</netty.tcnative.version>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-transport-native-epoll</artifactId>
<version>${netty.version}</version>
<classifier>${epoll.os}</classifier>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative</artifactId>
<version>${netty.tcnative.version}</version>
<classifier>${epoll.os}</classifier>
</dependency>
Right now I’m playing with
-Dio.netty.leakDetectionLevel=advanced
-Dio.netty.noPreferDirect=true
-Dio.netty.allocator.type=unpooled
-Dio.netty.maxDirectMemory=0
to find out working settings. I’ll update ticket with additional info if any.
Unfortunately I wasn’t able to reproduce this issue on QA env. Please let me know if you need more info.
Issue Analytics
- State:
- Created 7 years ago
- Comments:123 (122 by maintainers)
With the help of @doom369 I was able to track down the change that is guilty for this “regression”. https://github.com/netty/netty/pull/6252 should fix this. The commit message of https://github.com/netty/netty/pull/6252 should give you a better idea what happened, so I will not repeat it here but to make it short its not a memory leak but just a change in how much memory is used when using our custom SSLEngine impl.
@rkapsi @doom369 can you please check the PR and let me know…
Thanks again to @doom369 for all the help tracking this down. Without you this would have been not possible or took a way longer.
@doom369 awesome let me try 😃