"Failed to mark a promise as failure because it has failed already" in logs under heavy load

Hi Team.

Our log monitoring solution found following stack trace in one of our webflux instances.

2022-03-10 07:19:27.393 WARN 1 — [ctor-http-nio-1] i.n.c.AbstractChannelHandlerContext : Failed to mark a promise as failure because it has failed already: DefaultChannelPromise@25c05b40(failure: io.netty.channel.StacklessClosedChannelException), unnotified cause: io.netty.channel.StacklessClosedChannelException at io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source) io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1 at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74) ~[netty-common-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138) ~[netty-common-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100) ~[netty-buffer-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.handler.codec.http.DefaultFullHttpResponse.release(DefaultFullHttpResponse.java:116) ~[netty-codec-http-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:90) ~[netty-common-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:872) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1367) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:764) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:767) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:767) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:767) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.70.Final.jar!/:4.1.70.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.70.Final.jar!/:4.1.70.Final] at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

This happened only once under heavy load of performance tests. I couldn’t find any other negative side effect like failed web request or application failure.

I found that underlying ‘StacklessClosedChannelException’ is thrown internally on each healthcheck call (which calls another service to get its status) in our k8s cluster but is never logged anywhere. Under heavy load it leaked out together with warning from ticket description.

Netty version

4.1.70.Final

JVM version (e.g. java -version)

Distroless docker jdk image: openjdk version “11.0.14” 2022-01-18 OpenJDK Runtime Environment (build 11.0.14+9-post-Debian-1deb11u1) OpenJDK 64-Bit Server VM (build 11.0.14+9-post-Debian-1deb11u1, mixed mode)

[violetagg]

@pjablonski44 Please open an issue for Spring Cloud Gateway and provide more information for the SCG version, use case etc.

[pjablonski44]

It seems that root cause in my particular case is the downstream service that has broken ssl certificate as I was able to intercept the same exception locally by calling this endpoint from browser with ssl validation. Postman without ssl validation does not cause to throw anything. Probably kubernetes is accepting that ssl issue in liveness/readiness probes but also is causing this exception to be thrown. This particular healthcheck endpoint is served by spring cloud gateway configuration which passes the /api/healthcheck to downstream’s /api/healthcheck over https. It might be webflux issue under heavy load.