Support configuring per-method retryPolicy for gRPC channel

Description

io.quarkus.grpc.runtime.supports.Channels is responsible for producing instances of io.grpc.Channel. Most of io.grpc.Channel properties are set from application.properties. But not all. I’m looking for a way to call io.grpc.ManagedChannelBuilder#defaultServiceConfig to configure retry policy. This is needed to make gRPC retry actually work.

Setting quarkus.grpc.clients."service-name".retry=true is not enough. It enables retry subsystem, but retry policy must be enabled and configured for each method individually.

See this question and its links for more details: https://stackoverflow.com/questions/65701091/how-to-get-grpcs-retry-mechanism-to-work-using-grpc-java-in-kubernetes-cluster

Also official docs for gRPC retry: https://github.com/grpc/proposal/blob/master/A6-client-retries.md

[mkouba]

There are couple of things we need to consider:

• The retry policy is part of the service config which is bound to a Channel instance via io.grpc.internal.AbstractManagedChannelImplBuilder.defaultServiceConfig()
• A Channel instance is created per client configured via GrpcClientConfiguration
• So in therory, a Channel instance can be shared for multiple services…

The tricky part is that if we want to configure the retry policy per method then the config properties would have include the method name, e.g. something like quarkus.grpc.clients.hello.method.SayHello.retry.max-attempts=2. But (1) properties names are very long and (2) even then we would not be able to distinguish methods of a specific service.

We can also support per-client configuration that would be applied to all methods of all services:

quarkus.grpc.clients.hello.retry.max-attempts=2
quarkus.grpc.clients.hello.retry.initial-backoff=0.5s
quarkus.grpc.clients.hello.retry.max-backoff=1s
quarkus.grpc.clients.hello.retry.backoff-multiplier=1
quarkus.grpc.clients.hello.retry.retryable-status-codes=UNAVAILABLE

[cescoffier]

Good question about the configuration. @michalszynkiewicz do we have some kind of support?

About @Retry in the next version it will work with method returning Uni. I don’t believe it will work for Multi. I said next version, as it needs Smallrye Fault Tolerance 5.x which will come with a Quarkus 2.0