Netty server -> client messages apparently dropped in 0.13.2+

I am observing some strange behavior on grpc 0.13.2 and above, on my Macbook Pro. Diagnostic information at the bottom of this issue. Essentially what seems to happen is that the call is correctly handled by the server, all the way through onComplete, but the client never receives any of the inbound frames. This happens for, say, 5% of calls. I’ve attached an example log snippet, including annotations indicating where the various callbacks are invoked, as determined by our logging interceptors. This doesn’t reproduce on 0.13.1, but is consistently reproducible on 0.13.2. Unfortunately I was unable to come up with a minimal reproduction. Any help is greatly appreciated! Thanks!

Client side:

*** ClientCall#start() ***

*** ClientCall#halfClose() ***

2016-05-06T16:41:33.658Z DEBUG [grpc-default-worker-ELG-4-2] {} io.grpc.~.NettyClientHandler -
----------------OUTBOUND--------------------
[id: 0x9d2d086a, L:/127.0.0.1:59638 - R:/127.0.0.1:59512] HEADERS: streamId=1113, headers=DefaultHttp2Headers[:authority: 127.0.0.1:59512, :path: /inscriptive.sample.SampleService/Create, :method: POST, :scheme: http, trace_id: HJzC_U3l, grpc-accept-encoding: gzip, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.5], streamDependency=0, weight=16, exclusive=false, padding=0, endStream=false
------------------------------------
2016-05-06T16:41:33.658Z DEBUG [grpc-default-worker-ELG-4-2] {} io.grpc.~.NettyClientHandler -
----------------OUTBOUND--------------------
[id: 0x9d2d086a, L:/127.0.0.1:59638 - R:/127.0.0.1:59512] DATA: streamId=1113, padding=0, endStream=true, length=409, bytes=00000001940a1b74733a616a76766766513370593636476b5f4732676f715052504e12f4020a3b436f6e73756d657220507269636520496e64657820666f7220...
------------------------------------

Server side:

2016-05-06T16:41:33.658Z DEBUG [grpc-default-worker-ELG-4-2] {} io.grpc.~.NettyServerHandler -
----------------INBOUND--------------------
[id: 0x50d7d3f3, L:/127.0.0.1:59512 - R:/127.0.0.1:59638] HEADERS: streamId=1113, headers=DefaultHttp2Headers[:authority: 127.0.0.1:59512, :path: /inscriptive.sample.SampleService/Create, :method: POST, :scheme: http, trace_id: HJzC_U3l, grpc-accept-encoding: gzip, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.5], streamDependency=0, weight=16, exclusive=false, padding=0, endStream=false
------------------------------------

*** server interception chain invoked ***

2016-05-06T16:41:33.658Z DEBUG [grpc-default-worker-ELG-4-2] {} io.grpc.~.NettyServerHandler -
----------------INBOUND--------------------
[id: 0x50d7d3f3, L:/127.0.0.1:59512 - R:/127.0.0.1:59638] DATA: streamId=1113, padding=0, endStream=true, length=409, bytes=00000001940a1b74733a616a76766766513370593636476b5f4732676f715052504e12f4020a3b436f6e73756d657220507269636520496e64657820666f7220...
------------------------------------

*** ServerCall.Listener#onHalfClose() ***

...

*** ServerCall#close(Status.OK) ***

2016-05-06T16:41:33.676Z DEBUG [grpc-default-worker-ELG-4-2] {} io.grpc.~.NettyServerHandler -
----------------OUTBOUND--------------------
[id: 0x50d7d3f3, L:/127.0.0.1:59512 - R:/127.0.0.1:59638] HEADERS: streamId=1113, headers=DefaultHttp2Headers[:status: 200, grpc-encoding: gzip, grpc-accept-encoding: gzip, content-type: application/grpc], streamDependency=0, weight=16, exclusive=false, padding=0, endStream=false
------------------------------------
2016-05-06T16:41:33.676Z DEBUG [grpc-default-worker-ELG-4-2] {} io.grpc.~.NettyServerHandler -
----------------OUTBOUND--------------------
[id: 0x50d7d3f3, L:/127.0.0.1:59512 - R:/127.0.0.1:59638] DATA: streamId=1113, padding=0, endStream=false, length=5, bytes=0000000000
------------------------------------
2016-05-06T16:41:33.676Z DEBUG [grpc-default-worker-ELG-4-2] {} io.grpc.~.NettyServerHandler -
----------------OUTBOUND--------------------
[id: 0x50d7d3f3, L:/127.0.0.1:59512 - R:/127.0.0.1:59638] HEADERS: streamId=1113, headers=DefaultHttp2Headers[grpc-status: 0], streamDependency=0, weight=16, exclusive=false, padding=0, endStream=true
------------------------------------

*** ServerCall.Listener#onComplete() ***

$ java -version
java version "1.8.0_74"
Java(TM) SE Runtime Environment (build 1.8.0_74-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.74-b02, mixed mode)

[jack:/Users/jack/src/inscriptive] (jack-grpc-14) >> uname -a
Darwin Jacks-MacBook-Pro.local 15.3.0 Darwin Kernel Version 15.3.0: Thu Dec 10 18:40:58 PST 2015; root:xnu-3248.30.4~1/RELEASE_X86_64 x86_64

[johnbcoughlin]

Kay, it looks like with latest netty, netty-all-4.1.0.Final-20160506.120802-47, the issue doesn’t appear. I’m not convinced I was hitting #1682, but it could easily have been the same underlying netty issue.

[ejona86]

@johnbcoughlin, are any of the other calls on the Channel doing cancellation (including deadline)? If so, you may be hitting #1682 which will be fixed with the next Netty release. It’s taken a little longer than we initially expected to get the fix out. We should maybe send out an email to the mailing list.