FutureStub hangs when wrong netty version is used

Please answer these questions before submitting your issue.

What version of gRPC are you using?

1.3.0

What JVM are you using (java -version)?

netty4.version

What did you do?

If possible, provide a recipe for reproducing the error.

1. Accidentally use netty-codec-http2 4.1.9.Final in your pom
2. Create a future stub
3. Call stub.someOperation().get()

What did you expect to see?

• The call should fail from the underlying ClassNotFoundException

What did you see instead?

• A log message is printed to the console indicating the ClassNotFoundException
• The call to .get() blocks indefinitely because the underlying future never returns
• If a timeout is used when calling .get(), the request always times out

This problem was especially hard to diagnose because our logger is configured to exclude log messages from libraries (legacy code), so the ClassNotFoundException was never surfaced.

Findings

• ChannelExecutor.drain() (line 89) eats all exceptions thrown when processing the outbound task queue when DelayedClientTransport switches from DelayedStream to NettyClientStream in DelayedClientTransport.newStream().
• GrpcHttp2HeadersDecoder.<init> fails with a ClassNotFoundException when trying to load io.netty.handler.codec.http2.internal.hpack.Decoder.
• Since the stream switch is itself is the task that fails, DelayedClientTransport buffers outbound tasks indefinitely, causing the request to hang.

Possible Mitigations

1. Propagate exceptions from ChannelExecutor.drain() in a “CompositeException” that aggregates all exceptions encountered. This will cause the client to explode, rather than logging and hoping someone is watching.
2. Fail conspicuously when starting gRPC with an incompatible netty version.

[ejona86]

In other words, do I need to add safeguards against the future remaining empty, or do we feel pretty good that the future will be resolved as long as no panic-level errors occur?

You should feel pretty good that the future will be resolved as long as no panic-level errors occur.

would moving my resource-release logic into an interceptor be more robust than using the future callbacks?

Should be about the same.

The ClientInterceptor doc specifies it is used to intercept calls before dispatch, whereas I’d need to hook into post-request-completion processing … not sure if that’s possible on the client side.

It’s possible by making your own ClientCall.Listener (extending ForwardingClientCallListener).

[zhangkun83]

This is to be resolved by #4023