CircuitBreaker `onResponseTrailers` callback delayed invocation with empty HTTP-trailers

Context

In my setup, Armeria + unary gRPC are used. On the client side, I decorate the ClientBuilder with CircuitBreaker and CircuitBreakerRule in order to set up circuit breaking based on gRPC status code (e.g. when gRPC status code != OK, count the request as a failure).

Initially I passed a callback to CircuitBreakerRule.builder().onResponseTrailers(...) and expect the callback to be invoked when HTTP-trailers are received so that I can extract gRPC status code from the HTTP-trailers. However, I realized that when the server side throws an exception, server side does not send HTTP trailers (according to gRPC protocol specification). Instead, gRPC protocol specification names the response “trailers-only” which is physically represented as HTTP headers (and there is no body nor HTTP-trailers). Therefore in this case, the callback passed to CircuitBreakerRule.builder().onResponseTrailers(...) does not get invoked until after the exception is thrown to the client stub invocation level. When the callback is invoked, the trailers is empty (which is expected because gRPC’s “trailers-only” response does not have HTTP-trailers).

Temporary workaround

I changed CircuitBreakerRule.builder().onResponseTrailers(...) to CircuitBreakerRule.builder().onResponseHeaders(...) and try to extract gRPC status from the response HTTP headers. It works for the case where the server throws an exception and the response contains only HTTP headers (“trailers-only” in gRPC’s terminology) with gRPC status code.

Issues

1. The temporary workaround only works for the circuit breaker to detect failures in trailers-only response. However, the circuit breaker is completely blind to detect any failure if the server side sends back a response with 3 parts, namely headers, request body, and trailers where trailers contain the gRPC status code (as documented in the gRPC protocol specification).

2. Correct me if I am wrong on this one: when a CircuitBreakerRule.builder() has 2 callbacks registered, one from onResponseTrailers(...) and the other from onResponseHeaders(...), the onResponseHeaders callback does not get invoked when HTTP headers are received. Instead, it is invoked when HTTP trailers are received or when Armeria claims that there will be no HTTP trailers. There is a flag requiresResponseTrailers on each CircuitBreakerRule to decide which registered callbacks should be invoked. This behavior makes adding both on-responseHeaders and on-responseTrailers callbacks not an option for me because the on-responseHeaders will be invoked too late so that the circuit breaker fails to short-circuit the next client stub invocation.

Proposed improvement

1. Change the behavior described in the above Issues # 2 so that when CircuitBreakerRule.builder() has both on-responseHeaders and on-responseTrailers callbacks registered, the on-responseHeader callback is invoked immediately when HTTP headers are received. IMHO it does not make sense semantically to wait for trailers and then invoke the on-responseHeader callback.
2. Make Armeria be aware of such “trailers-only” response. The ideal behavior IMO would be: when Armeria receives HTTP headers, it can tell that these headers are gRPC “trails-only” and there will be no body nor HTTP trailers coming in later. Then Armeria invoke the on-responseHeaders (with received HTTP headers) and on-responseTrailers (with nothing) immediately. Because Armeria supports gRPC as a first-class citizen, it is reasonable (to some extent at least) to make Armeria be able to interpret gRPC trailers-only.

Let me know if I misunderstood anything and what you think. Thanks!

[ikhoon]

@ikhoon if you can point me to the code showing the below behavior, that’d be great!

If you debug the following lines with breakpoints, we can check that a failed response is completed as the trailers-only is received. https://github.com/line/armeria/blob/876339437d566cd74e0f87a090430441b3da1f4e/core/src/main/java/com/linecorp/armeria/client/Http2ResponseDecoder.java#L215-L217 https://github.com/line/armeria/blob/daae15d426ffb2581fdff9f10f16ec69f90b7a36/core/src/main/java/com/linecorp/armeria/common/logging/DefaultRequestLog.java#L412 https://github.com/line/armeria/blob/a9592f49c501beff47bdb02788d6e79009c51ffc/core/src/main/java/com/linecorp/armeria/client/circuitbreaker/CircuitBreakerClient.java#L309

[ikhoon]

Print statement output with both onResponseHeaders and onResponseTrailers

As you can see, +++ onResponseHeaders called happened after the first call (after calling foo…) and even after the start of the second call — before calling foo(): attempt: 1.

We don’t guarantee onResponseTrailers() should be called before foo() returns or raises an exception. Because CircuitBreakerClient asynchronously checks the responses and reports the CircuitBreakerDecision to the CircuitBreaker.

• If only onResponseHeaders() is registered, the onResponseHeaders() may be called before the foo() is returned. Because it only doesn’t need to wait for a response to be closed.
• If both onResponseHeaders() and onResponseTrailers() are registered, it is possible to call the callbacks after the call raises an error. Because the call may be finished immediately by here and the callbacks would be called later by here.

Trailers-only response itself does NOT contain END_STREAM HTTP frame.

It is possible. But I guess you used AbstractUnaryGrpcService. If so, END_STREAM is always sent with trailers-only response. https://github.com/line/armeria/blob/2343b6ebb04ef8d8da9fb5e25663a11f626a3939/core/src/main/java/com/linecorp/armeria/server/AbstractHttpResponseHandler.java#L148