Questions about exception propagation in Armeria gRPC client with a RetryingClient decorator

Hi there! I have 2 questions about how exception propagation behavior in Armeria gRPC client with a RetryingClient decorator. It seems to me that there could be a bug.

Setup

Armeria version: 1.16.0

Client setup

I build a gRPC client stub with a ClientBuilder and I add 3 decorators to the ClientBuilder as shown below:

// Please note that "Site A/B/C/D" mentioned in the comment below will be referred in later discussion.

// Decorator 1
clientBuilder.decorator((delegate, ctx, req) => {
    val httpResponse = delegate.execute(ctx, req)
    httpResponse
        .mapHeaders(respHeaders => {
             // Site A
             // 1. create an exception based on the response header content.
             // 2. throw the created exception.
        }
})

// Decorator 2
clientBuilder.decorator((delegate, ctx, req) => {
    ctx.log() // ClientRequestContext
      .whenComplete()
      .thenAccept(requestLog => {
          val exception = requestLog.responseCause()
          // The above `exception` is always null except in the case where
          // the last retry attempt fails.
          // Site B
       })
    delegate.execute(ctx, req)
})

// Decorator 3
val retryDecorator = RetryingClient.newDecorator(createRetryConfig())
clientBuilder.decorator(retryDecorator)

def createRetryConfig(): RetryConfig = {
   // ...
   val retryRuleBuilder = RetryRule.builder()
   retryRuleBuilder.onException((_: ClientRequestContext, cause: Throwable) => {
     // This callback is NEVER invoked.
     // Site C
   })
   retryRuleBuilder.onResponseTrailers((ctx, trailers) => {
     // This callback is invoked. The retry decision is made based on `trailers`
     // Site D
   })
   // Create a RetryConfig with the above `retryRuleBuilder` and return it.
}

Test setup

I wrote a test where a client with above decorators send one request to a server which always fails the requests and populates the response header with some error information. When the client receives the response, I expect the decorator 1’s httpResponse.mapHeaders callback (Site A) is invoked and an exception is created and thrown there. Then I expect the client to retry up to 3 times and eventually fail (because the server always fails the request and triggers an exception on the client side).

Observations

1. Retry kicked in and 3 requests were sent sequentially. All 3 attempts failed that lead to client stub invocation failure. This behavior is expected 😃
2. The order of invocation during each request attempt: Site A --> Site D --> Site B. Note that Site C is never invoked.
3. In the first 2 request attempts at Site B, requestLog.responseCause() == null. However, in the 3rd/final request attempt at Site B, requestLog.responseCause() == exception_thrown_at_Site A.

Questions

1. For observation 2, why does Site C never invoked? Is it a bug or I did something wrong?
2. For observation 3, is this the expected/correct behavior? Specifically, does Armeria intentionally try to hide the exception from Site B (the ctx.log().whenComplete() callback) until the max retry attempt is reached? If it is intentional, could you share the rationale behind this design?

[minwoox]

1.16.1 has been released. 😉 https://repo.maven.apache.org/maven2/com/linecorp/armeria/armeria-bom/1.16.1/ https://github.com/line/armeria/commits/1cca43dec3b7cb84dc4131e228603dcf3b9caa7c

[Lincong]

If you want, I can release a patch version.

That’d be great! Here is the reason why it would be super helpful for us:

We are on 1.16.0 and my team currently does not have the resource/bandwidth to upgrade to the newest release version of Armeria. So, a patch fixing the issue in 1.16.0 would be the best.

Thanks! @minwoox