Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

io.netty.util.IllegalReferenceCountException on random HTTP requests

See original GitHub issue

We crawl a lot of websites and from time to time get the following exception:

io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1 (class io.netty.util.IllegalReferenceCountException)
io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100)
io.netty.handler.codec.http.DefaultFullHttpResponse.release(DefaultFullHttpResponse.java:115)
io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:88)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:90)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.base/java.lang.Thread.run(Thread.java:834)

It happens randomly. That is: The exception may be thrown when crawling some URL, but it will not occur when crawling the URL again.

Expected behavior

References are counted correctly

Actual behavior

References are counted incorrectly, which causes an exception to be thrown.

Steps to reproduce

Difficult. Happens randomly.

Minimal yet complete reproducer code (or URL to code)

Not available.

Netty version

4.1.42.Final

JVM version (e.g. `java -version`)

openjdk version “11.0.4” 2019-07-16 OpenJDK Runtime Environment (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3) OpenJDK 64-Bit Server VM (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3, mixed mode, sharing)

OS version (e.g. `uname -a`)

Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-62-generic x86_64)

Issue Analytics

State:
Created 4 years ago
Reactions:4
Comments:10 (3 by maintainers)

Top GitHub Comments

6reactions

m-zaicommented, Feb 17, 2020

We have the same issue using spring-boot-starter-reactor-netty 2.2.4.RELEASE. However, it occurs only on large data fetch. Do you have any idea about next release date of a bug free version ?

1reaction

carryxyhcommented, Dec 23, 2019

Seems this issue’s solution will help u: https://github.com/netty/netty/issues/9229

I submitted a pull request to implement this idea: https://github.com/netty/netty/pull/9892

In the future, IllegalReferenceCountException will be able to record the recent access records like leak detection… But currently maybe there is no good way to easily find the root cause…

Top Results From Across the Web

Webflux - How to prevent an IllegalReferenceCountException ...

Fix- I was passing the incoming http request org.springframework.core.io.buffer.DataBuffer directly to the WebClient request body.

server/netty: IllegalReferenceCountException : KTOR-1801

I got this random error out of nowhere. I'm not even using the Ktor server. It's just random bots on the web making...

Micronaut Framework/questions - micronautfw/questions - Gitter

io.netty.util.IllegalReferenceCountException: refCnt: 0 at io.netty.buffer. ... When I try to upload a large file using the Micronaut http client, ...

Serialized Form (Netty API Reference (4.1.85.Final))

Package io.netty.handler.codec.http.multipart ... Initialization flag to permit calls to setSeed to succeed only while executing the Random constructor.

Index (Netty API Reference (4.1.59.Final)) - javadoc.io

ACCESS_CONTROL_REQUEST_METHOD - Static variable in class io.netty.handler.codec.http.HttpHeaders.Names. Deprecated. "Access-Control-Request-Method".