In some use cases the execution hangs when HTTP/2 response is returned from an arbitrary thread

Expected behavior

Data is served successfully when HTTP/2 with prior knowledge.

Actual behavior

In some use cases when HTTP/2 with prior knowledge only headers are returned and the execution hangs. Below you can find a reproducible scenario. However the use case is the following:

1. The response will be returned from an arbitrary thread.
2. ctx.writeAndFlush is used in order to write the headers.
3. When the headers are successfully written and flushed, a notification is send and the data will be written using ctx.write(). This happens in one and the same thread and it is triggered as a result of a successful headers’ flush operation - https://github.com/netty/netty/blob/netty-4.1.45.Final/codec-http2/src/main/java/io/netty/handler/codec/http2/AbstractHttp2StreamChannel.java#L1039
4. The flush operation for the data will be scheduled so there is no immediate flush operation.
5. After exiting https://github.com/netty/netty/blob/netty-4.1.45.Final/codec-http2/src/main/java/io/netty/handler/codec/http2/AbstractHttp2StreamChannel.java#L1041 writeDoneAndNoFlush is set to false and thus the scheduled flushed operation will not be executed which leads to the situation when the data will not be send to the client.

When moving writeDoneAndNoFlush before the flush the scenario is working but I’m not sure whether such change will not break other scenarios:

diff --git a/codec-http2/src/main/java/io/netty/handler/codec/http2/AbstractHttp2StreamChannel.java b/codec-http2/src/main/java/io/netty/handler/codec/http2/AbstractHttp2StreamChannel.java
index 2b894cfabe..aecefb1288 100644
--- a/codec-http2/src/main/java/io/netty/handler/codec/http2/AbstractHttp2StreamChannel.java
+++ b/codec-http2/src/main/java/io/netty/handler/codec/http2/AbstractHttp2StreamChannel.java
@@ -1035,11 +1035,11 @@ abstract class AbstractHttp2StreamChannel extends DefaultAttributeMap implements
                 // There is nothing to flush so this is a NOOP.
                 return;
             }
-            try {
-                flush0(parentContext());
-            } finally {
-                writeDoneAndNoFlush = false;
-            }
+            //try {
+            writeDoneAndNoFlush = false;
+            flush0(parentContext());
+            //} finally {
+            //}
         }
 
         @Override

Here is the stack:

lambda$channelRead$0:98, TestServer$3$1
operationComplete:-1, 759257819 (TestServer$3$1$$Lambda$4)
notifyListener0:577, DefaultPromise (io.netty.util.concurrent)
notifyListenersNow:551, DefaultPromise (io.netty.util.concurrent)
notifyListeners:490, DefaultPromise (io.netty.util.concurrent)
setValue0:615, DefaultPromise (io.netty.util.concurrent)
setSuccess0:604, DefaultPromise (io.netty.util.concurrent)
setSuccess:96, DefaultPromise (io.netty.util.concurrent)
setSuccess:78, DefaultChannelPromise (io.netty.channel)
setSuccess:73, DefaultChannelPromise (io.netty.channel)
writeComplete:992, AbstractHttp2StreamChannel$Http2ChannelUnsafe (io.netty.handler.codec.http2)
access$2000:582, AbstractHttp2StreamChannel$Http2ChannelUnsafe (io.netty.handler.codec.http2)
operationComplete:967, AbstractHttp2StreamChannel$Http2ChannelUnsafe$4 (io.netty.handler.codec.http2)
operationComplete:964, AbstractHttp2StreamChannel$Http2ChannelUnsafe$4 (io.netty.handler.codec.http2)
notifyListener0:577, DefaultPromise (io.netty.util.concurrent)
notifyListenersNow:551, DefaultPromise (io.netty.util.concurrent)
notifyListeners:490, DefaultPromise (io.netty.util.concurrent)
setValue0:615, DefaultPromise (io.netty.util.concurrent)
setSuccess0:604, DefaultPromise (io.netty.util.concurrent)
trySuccess:104, DefaultPromise (io.netty.util.concurrent)
trySuccess:84, DefaultChannelPromise (io.netty.channel)
tryPromise:382, Http2CodecUtil$SimpleChannelPromiseAggregator (io.netty.handler.codec.http2)
trySuccess:349, Http2CodecUtil$SimpleChannelPromiseAggregator (io.netty.handler.codec.http2)
trySuccess:261, Http2CodecUtil$SimpleChannelPromiseAggregator (io.netty.handler.codec.http2)
trySuccess:48, PromiseNotificationUtil (io.netty.util.internal)
safeSuccess:717, ChannelOutboundBuffer (io.netty.channel)
remove:272, ChannelOutboundBuffer (io.netty.channel)
removeBytes:352, ChannelOutboundBuffer (io.netty.channel)
doWrite:431, NioSocketChannel (io.netty.channel.socket.nio)
flush0:930, AbstractChannel$AbstractUnsafe (io.netty.channel)
flush0:354, AbstractNioChannel$AbstractNioUnsafe (io.netty.channel.nio)
flush:897, AbstractChannel$AbstractUnsafe (io.netty.channel)
flush:1372, DefaultChannelPipeline$HeadContext (io.netty.channel)
invokeFlush0:748, AbstractChannelHandlerContext (io.netty.channel)
invokeFlush:740, AbstractChannelHandlerContext (io.netty.channel)
flush:726, AbstractChannelHandlerContext (io.netty.channel)
flush:189, Http2ConnectionHandler (io.netty.handler.codec.http2)
invokeFlush0:748, AbstractChannelHandlerContext (io.netty.channel)
invokeFlush:740, AbstractChannelHandlerContext (io.netty.channel)
flush:726, AbstractChannelHandlerContext (io.netty.channel)
flush0:1089, AbstractHttp2StreamChannel (io.netty.handler.codec.http2)
flush:1039, AbstractHttp2StreamChannel$Http2ChannelUnsafe (io.netty.handler.codec.http2)
flush:1372, DefaultChannelPipeline$HeadContext (io.netty.channel)
invokeFlush0:748, AbstractChannelHandlerContext (io.netty.channel)
invokeFlush:740, AbstractChannelHandlerContext (io.netty.channel)
flush:726, AbstractChannelHandlerContext (io.netty.channel)
flush:127, ChannelDuplexHandler (io.netty.channel)
invokeFlush0:748, AbstractChannelHandlerContext (io.netty.channel)
invokeWriteAndFlush:763, AbstractChannelHandlerContext (io.netty.channel)
run:1089, AbstractChannelHandlerContext$WriteTask (io.netty.channel)
safeExecute:164, AbstractEventExecutor (io.netty.util.concurrent)
runAllTasks:472, SingleThreadEventExecutor (io.netty.util.concurrent)
run:500, NioEventLoop (io.netty.channel.nio)
run:989, SingleThreadEventExecutor$4 (io.netty.util.concurrent)
run:74, ThreadExecutorMap$2 (io.netty.util.internal)
run:30, FastThreadLocalRunnable (io.netty.util.concurrent)
run:748, Thread (java.lang)

Steps to reproduce

1. Start the provided TestServer
2. In the terminal execute curl --http2-prior-knowledge http://localhost:8080/ -v
3. You should be able to see that the headers are returned and the execution hangs

Minimal yet complete reproducer code (or URL to code)

https://gist.github.com/violetagg/1b8d55276e3a00a5d153c6a5608a4251

Netty version

4.1.45.Final and 4.1.46.Final-SNAPSHOT

JVM version (e.g. java -version)

java version “1.8.0_231” Java™ SE Runtime Environment (build 1.8.0_231-b11) Java HotSpot™ 64-Bit Server VM (build 25.231-b11, mixed mode)

OS version (e.g. uname -a)

Darwin Violetas-MacBook-Pro.local 19.3.0 Darwin Kernel Version 19.3.0: Thu Jan 9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64 x86_64

[normanmaurer]

@violetagg thank again for the report and reproducer. Your “fix” also looks correct. I am working on a unit test now and will hopefully be able to open a PR later today.

[kiranmaiem]

thanks