Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

ByteBuf leak

See original GitHub issue

I have upgraded to the latest version (2.0.0-SNAPSHOT) and I am now getting an OutOfDirectMemoryError after some time of normal usage.

The server is configured to monitor the applications registered to our eureka server.

Here is the stacktrace:

io.netty.handler.codec.EncoderException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 251658247, max: 257425408)
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:304) ~[netty-handler-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:291) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:465) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:191) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1051) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
        at reactor.ipc.netty.FutureMono.lambda$deferFuture$0(FutureMono.java:68) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
        at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:134) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
        at reactor.core.publisher.FluxConcatIterable$ConcatIterableSubscriber.onComplete(FluxConcatIterable.java:141) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
        at reactor.core.publisher.FluxConcatIterable.subscribe(FluxConcatIterable.java:60) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
        at reactor.core.publisher.MonoSourceFlux.subscribe(MonoSourceFlux.java:47) ~[reactor-core-3.1.5.RELEASE.jar!/:3.1.5.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:380) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
        at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:501) ~[reactor-netty-0.7.5.RELEASE.jar!/:0.7.5.RELEASE]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[netty-transport-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]
Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 251658247, max: 257425408)
        at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:594) ~[netty-common-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:764) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:740) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:214) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:113) ~[netty-buffer-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:92) ~[netty-codec-http-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.22.Final.jar!/:4.1.22.Final]
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.22.Final.jar!/:4.1.22.Final]
        ... 37 common frames omitted

Anyone else with this problem?

Issue Analytics

State:
Created 6 years ago
Comments:28 (10 by maintainers)

Top GitHub Comments

2reactions

n0mercommented, Mar 18, 2018

@joshiste what about releasing RC1 , since 90% of issues are fixed for 2.0.0?

1reaction

joshistecommented, Mar 23, 2018

I finally could track down the issue. -Dio.netty.leakDetectionLevel=paranoid helped. It turned out that the responses from the WebClient weren’t always consumed and therefore the underlying buffers weren’t released.

Top Results From Across the Web

LEAK: ByteBuf.release() was not called before it's garbage ...

You need to call release on the ByteBuf rather than clear in your finally block, otherwise the reference counted buffer is not de-allocated...

LEAK: ByteBuf.release() was not called before it's garbage ...

Expected behavior No leak Actual behavior LEAK: ByteBuf.release() was not called before it's garbage-collected LEAK: ByteBuf.release() was ...

A Netty ByteBuf Memory Leak Story and the Lessons Learned

We were chasing a memory leak while refactoring our log receiver. After a major refactoring, we noticed a gradual decrease of free memory...

Reference counted objects - Netty.docs

ByteBuf is the most notable type taking advantage of reference counting to improve ... LEAK: ByteBuf.release() was not called before it's garbage-collected.

LEAK: ByteBuf.release() was not called before it's garbage ...

After some time collecting data from a modbus TCP device we get the error: 16:29:50.228 ERROR ResourceLeakDetector: LEAK: ByteBuf.release() ...