Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Netty reports LEAK while using reactor (Rsocket) with Sleuth

See original GitHub issue

Describe the bug While using spring cloud sleuth with Rsocket I am getting Netty memory leak errors. If I make sleuth disabled - there are no leak errors reported. Seems the leak source is org.springframework.cloud.sleuth.instrument.rsocket.PayloadUtils.payload(PayloadUtils.java:59) which copies the metamodel payload in my case.

Versions

  • spring-cloud-sleuth: 3.1.0
  • spring-boot: 2.6.x (tried with 2.6.3 and prior)
  • reactor-core: 3.4.14
  • rsocket-core: 1.1.1

Details Here is the more detailed logs from netty:`

12:05:19.633 ERROR [7a40f25cc1a93552,f54a4a5d908cc677] [boundedElastic-9][ResourceLeakDetector] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.

Recent access records:
#1: io.rsocket.util.DefaultPayload.create(DefaultPayload.java:107)
             org.springframework.cloud.sleuth.instrument.rsocket.PayloadUtils.payload(PayloadUtils.java:59)
             org.springframework.cloud.sleuth.instrument.rsocket.PayloadUtils.cleanTracingMetadata(PayloadUtils.java:49)
org.springframework.cloud.sleuth.instrument.rsocket.TracingResponderRSocketProxy.requestResponse(TracingResponderRSocketProxy.java:95)
                io.rsocket.core.RSocketResponder.requestResponse(RSocketResponder.java:123)
                io.rsocket.core.RSocketResponder.handleRequestResponse(RSocketResponder.java:355)
                io.rsocket.core.RSocketResponder.handleFrame(RSocketResponder.java:209)
                reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:160)
io.rsocket.core.ClientServerInputMultiplexer$InternalDuplexConnection.onNext(ClientServerInputMultiplexer.java:248)
                io.rsocket.core.ClientServerInputMultiplexer.onNext(ClientServerInputMultiplexer.java:129)
                io.rsocket.core.ClientServerInputMultiplexer.onNext(ClientServerInputMultiplexer.java:48)
                io.rsocket.core.SetupHandlingDuplexConnection.onNext(SetupHandlingDuplexConnection.java:118)
                io.rsocket.core.SetupHandlingDuplexConnection.onNext(SetupHandlingDuplexConnection.java:19)
                reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120)
                reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:364)
                reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
                reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:594)
                reactor.netty.http.server.WebsocketServerOperations.onInboundNext(WebsocketServerOperations.java:169)
                reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
                io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
                io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
                io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371)
                io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234)
                io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283)
  io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
                io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
                io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
                io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
                io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
                io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
                io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
                io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
                io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
                io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
                io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
                io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
                java.lang.Thread.run(Thread.java:748)
#2:
      io.netty.buffer.AdvancedLeakAwareCompositeByteBuf.readBytes(AdvancedLeakAwareCompositeByteBuf.java:476)
        io.netty.buffer.AdvancedLeakAwareCompositeByteBuf.readBytes(AdvancedLeakAwareCompositeByteBuf.java:36)
                io.rsocket.util.DefaultPayload.toBytes(DefaultPayload.java:120)
                io.rsocket.util.DefaultPayload.create(DefaultPayload.java:103)
                org.springframework.cloud.sleuth.instrument.rsocket.PayloadUtils.payload(PayloadUtils.java:59)
             org.springframework.cloud.sleuth.instrument.rsocket.PayloadUtils.cleanTracingMetadata(PayloadUtils.java:49)
org.springframework.cloud.sleuth.instrument.rsocket.TracingResponderRSocketProxy.requestResponse(TracingResponderRSocketProxy.java:95)
                io.rsocket.core.RSocketResponder.requestResponse(RSocketResponder.java:123)
                io.rsocket.core.RSocketResponder.handleRequestResponse(RSocketResponder.java:355)
                io.rsocket.core.RSocketResponder.handleFrame(RSocketResponder.java:209)
                reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:160)
io.rsocket.core.ClientServerInputMultiplexer$InternalDuplexConnection.onNext(ClientServerInputMultiplexer.java:248)
                io.rsocket.core.ClientServerInputMultiplexer.onNext(ClientServerInputMultiplexer.java:129)
                io.rsocket.core.ClientServerInputMultiplexer.onNext(ClientServerInputMultiplexer.java:48)
                io.rsocket.core.SetupHandlingDuplexConnection.onNext(SetupHandlingDuplexConnection.java:118)
                io.rsocket.core.SetupHandlingDuplexConnection.onNext(SetupHandlingDuplexConnection.java:19)
                reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120)
                reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:364)
                reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
                reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:594)
                reactor.netty.http.server.WebsocketServerOperations.onInboundNext(WebsocketServerOperations.java:169)
                reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
                io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
                io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
                io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371)
                io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234)
                io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283)
  io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
                io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
                io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
                io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
                io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
                io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
                io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
                io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
                io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
                io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
                io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
                io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
                java.lang.Thread.run(Thread.java:748)
#3:
                org.springframework.cloud.sleuth.instrument.rsocket.PayloadUtils.payload(PayloadUtils.java:59)
             org.springframework.cloud.sleuth.instrument.rsocket.PayloadUtils.cleanTracingMetadata(PayloadUtils.java:49)
org.springframework.cloud.sleuth.instrument.rsocket.TracingResponderRSocketProxy.requestResponse(TracingResponderRSocketProxy.java:95)
                io.rsocket.core.RSocketResponder.requestResponse(RSocketResponder.java:123)
                io.rsocket.core.RSocketResponder.handleRequestResponse(RSocketResponder.java:355)
                io.rsocket.core.RSocketResponder.handleFrame(RSocketResponder.java:209)
                reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:160)
io.rsocket.core.ClientServerInputMultiplexer$InternalDuplexConnection.onNext(ClientServerInputMultiplexer.java:248)
                io.rsocket.core.ClientServerInputMultiplexer.onNext(ClientServerInputMultiplexer.java:129)
                io.rsocket.core.ClientServerInputMultiplexer.onNext(ClientServerInputMultiplexer.java:48)
                io.rsocket.core.SetupHandlingDuplexConnection.onNext(SetupHandlingDuplexConnection.java:118)
                io.rsocket.core.SetupHandlingDuplexConnection.onNext(SetupHandlingDuplexConnection.java:19)
                reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120)
                reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:364)
                reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
                reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:594)
                reactor.netty.http.server.WebsocketServerOperations.onInboundNext(WebsocketServerOperations.java:169)
                reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
                io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
                io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
                io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371)
                io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234)
                io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283)
  io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
                io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
                io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
                io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
                io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
                io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
                io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
                io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
                io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
                io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
                io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
                io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
                java.lang.Thread.run(Thread.java:748)
#4:

io.netty.buffer.AdvancedLeakAwareCompositeByteBuf.addComponents(AdvancedLeakAwareCompositeByteBuf.java:926)
io.rsocket.metadata.CompositeMetadataCodec.encodeAndAddMetadataWithCompression(CompositeMetadataCodec.java:266)
             org.springframework.cloud.sleuth.instrument.rsocket.PayloadUtils.cleanTracingMetadata(PayloadUtils.java:45)
org.springframework.cloud.sleuth.instrument.rsocket.TracingResponderRSocketProxy.requestResponse(TracingResponderRSocketProxy.java:95)
                io.rsocket.core.RSocketResponder.requestResponse(RSocketResponder.java:123)
                io.rsocket.core.RSocketResponder.handleRequestResponse(RSocketResponder.java:355)
                io.rsocket.core.RSocketResponder.handleFrame(RSocketResponder.java:209)
                reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:160)
io.rsocket.core.ClientServerInputMultiplexer$InternalDuplexConnection.onNext(ClientServerInputMultiplexer.java:248)
                io.rsocket.core.ClientServerInputMultiplexer.onNext(ClientServerInputMultiplexer.java:129)
                io.rsocket.core.ClientServerInputMultiplexer.onNext(ClientServerInputMultiplexer.java:48)
                io.rsocket.core.SetupHandlingDuplexConnection.onNext(SetupHandlingDuplexConnection.java:118)
                io.rsocket.core.SetupHandlingDuplexConnection.onNext(SetupHandlingDuplexConnection.java:19)
                reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120)
                reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:364)
                reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
                reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:594)
                reactor.netty.http.server.WebsocketServerOperations.onInboundNext(WebsocketServerOperations.java:169)
                reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
                io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
                io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
                io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371)
                io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234)
                io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283)
  io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
                io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
                io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
                io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
                io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
                io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
                io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
                io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
                io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
                io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
                io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
                io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
                java.lang.Thread.run(Thread.java:748)
Created at:
                io.netty.buffer.AbstractByteBufAllocator.compositeDirectBuffer(AbstractByteBufAllocator.java:224)
                io.netty.buffer.AbstractByteBufAllocator.compositeDirectBuffer(AbstractByteBufAllocator.java:219)
                io.netty.buffer.AbstractByteBufAllocator.compositeBuffer(AbstractByteBufAllocator.java:194)
             org.springframework.cloud.sleuth.instrument.rsocket.PayloadUtils.cleanTracingMetadata(PayloadUtils.java:42)
org.springframework.cloud.sleuth.instrument.rsocket.TracingResponderRSocketProxy.requestResponse(TracingResponderRSocketProxy.java:95)
                io.rsocket.core.RSocketResponder.requestResponse(RSocketResponder.java:123)
                io.rsocket.core.RSocketResponder.handleRequestResponse(RSocketResponder.java:355)
                io.rsocket.core.RSocketResponder.handleFrame(RSocketResponder.java:209)
                reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:160)
io.rsocket.core.ClientServerInputMultiplexer$InternalDuplexConnection.onNext(ClientServerInputMultiplexer.java:248)
                io.rsocket.core.ClientServerInputMultiplexer.onNext(ClientServerInputMultiplexer.java:129)
                io.rsocket.core.ClientServerInputMultiplexer.onNext(ClientServerInputMultiplexer.java:48)
                io.rsocket.core.SetupHandlingDuplexConnection.onNext(SetupHandlingDuplexConnection.java:118)
                io.rsocket.core.SetupHandlingDuplexConnection.onNext(SetupHandlingDuplexConnection.java:19)
                reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120)
                reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:364)
                reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:404)
                reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:594)
                reactor.netty.http.server.WebsocketServerOperations.onInboundNext(WebsocketServerOperations.java:169)
                reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
                io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
                io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
                io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371)
                io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234)
                io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283)
  io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
                io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
                io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
                io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
                io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
                io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
                io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
                io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
                io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
                io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
                io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
                io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
                io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
                java.lang.Thread.run(Thread.java:748)

: 1 leak records were discarded because they were duplicates

Sample memory-leak.zip or from github https://github.com/progys/sleuth-memory-leak

To run this app extract it and go to content dir of extracted and execute: mvnw test

Observe that test ran and finished successfully, but log contains Netty leak reports: image

Now go to src/main/resources/application.yml and disable sleuth completely by setting config value to false:

spring:
   sleuth:
     enabled: false

Re-run same command mvnw test and observe that log does not contain any Netty leak reports: image

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:2
  • Comments:10 (7 by maintainers)

github_iconTop GitHub Comments

1reaction
marcingrzejszczakcommented, Feb 7, 2022

looking at @OlegDokuka for help 😬

1reaction
progyscommented, Jan 25, 2022

Updated description with the minimal sample app and instructions how to run it. Let me know if you can run it. It depends on JVM a bit as it tries to force garbage collection by using System.gc(), but with my default Java 11.0.11 it logs memory leak each time, so hopefully it works on your setup too. If not - let me know and maybe I can create some better example.

Test scenario:

  • Netty based rsocket server listens for incoming rocket request using request/reply pattern (even though also reproducible with channel or stream, so seems not related);
  • Client sends request/reply with empty json object (can be any payload);
  • Server replies with plain text;
  • Client receives text;
  • Test triggers System.gc();
  • Netty leak report is printed to the console;
  • Test finishes and client closes the connection;

That said - I do not know if this actually creates a memory leak which leads to OOM, but if I do not change anything, just disable sleuth - there are no memory leak reports printed in the logs while running the same test. Exactly same behavior is observed in the real PROD like application.

To see if this is a real LEAK or just some noise this test can be easily changed to send huge amount of data in the client request and memory consumption can be monitored to see if memory is released from the server after request is finished.

Thanks for your efforts in analyzing this!

Read more comments on GitHub >

github_iconTop Results From Across the Web

reactor/reactor-netty - Gitter
I'm using reactor-netty 0.9.1.RELEASE and I want to monitor how many connections are actively being used by a fixed reactor.netty.resources.ConnectionProvider .
Read more >
Spring Cloud sleuth memory leak issue - Stack Overflow
Problem is whenever I am using spring cloud sleuth dependecny with @enablecaching enable my heapdumps are showing my memory leak at org.
Read more >
Spring Cloud 2021.0.2 Has Been Released
On behalf of the community, I am pleased to announce that the Service Release 2 of the Spring Cloud 2021.0 Release Train is...
Read more >
Java Weekly, Issue 323 - Baeldung
An intro to the RSocket messaging protocol for microservices, and a good write-up on ... CVE Reports Published for Reactor Netty [spring.io].
Read more >
Reactor Netty Reference Guide
Get in touch with the community on Gitter. Ask a question on stackoverflow.com at reactor-netty . Report bugs in Github issues. The repository ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Allow changing the format of the traceIDs so they are compatible with AWS X-Ray

Next page

Headers not always propagated to MDC (with sample)