Endpoint 'gRPC - Unimplemented service'

I have a service that is not behaving as it should. While troubleshooting it, I noticed this debug output:

Endpoint ‘gRPC - Unimplemented service’ with route pattern ‘{unimplementedService}/{unimplementedMethod}’ is valid for the request path … (the path of the non-working service).

What does this mean? I register my Grpc-service using endpoints.MapGrpcService<InforARIService>();

The actual issue I was troubleshooting is that the Grpc-service is invoked, does it’s work correctly and returns a class with two string properties. However, on the wire the response is empty (so the client gets a serialization error):

HTTP/1.1 200 OK
Content-Length: 0
Content-Type: application/grpc-web
Date: Fri, 08 Apr 2022 09:23:39 GMT
Server: Kestrel
Request-Context: appId=cid-v1:87cb2164-3464-497d-a7b4-80e9471bf6c3
Grpc-Status: 0
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000

[halter73]

Thanks for the simple repro app. You can fix your issue by passing through response flushes immediately. This way the gRPC trailer-writing logic knows when the response has already started and does not incorrectly consider the response “Trailers-Only” adds the grpc-status: 0 trailer as a normal header sent before the response body instead of after the response body:

https://github.com/grpc/grpc-dotnet/blob/197b43bbf784603d3f2e286f33e43f8b1dbdef72/src/Grpc.AspNetCore.Server/Internal/GrpcProtocolHelpers.cs#L156-L175

The workaround:

class ResponseBufferingStream : MemoryStream
{
    private readonly Stream _originalResponseStream;

    public ResponseBufferingStream(Stream originalResponseStream)
    {
        _originalResponseStream = originalResponseStream;
    }

    public override Task FlushAsync(CancellationToken cancellationToken)
    {
        return _originalResponseStream.FlushAsync(cancellationToken);
    }
}

And then the following change to RequestGrabber.cs

- using var replacementResponseBody = new MemoryStream();
+ using var replacementResponseBody = new ResponseBufferingStream(originalResponseBody);

Even though ResponseBufferingStream doesn’t actually flush any bytes to the response body that may have already been written to the MemoryStream, flushing the HttpResponse.Body always flushes the response headers which gives HttpResponse.HasStarted its proper value of true which is what’s important here.

I did not override all the methods an app could use to try to flush the response, but I know the gRPC logic calls HttpResponse.StartAsync() which calls through to this when you replace the HttpResponse.Body stream.

I think this should probably be fixed in gRPC. If the gRPC logic independently tracked if it has written to the body instead of using response.HasStarted, the simpler MemoryStream response body replacement could still work like it does for most other kinds of requests.

---

At first, I thought this was going to a case of not flushing HttpResponse.BodyWriter before copying the response body. I was going to suggest changing the following change to RequestGrabber:

// Continue processing (additional middleware, controller, etc.)
await next(context);
+ await response.CompleteAsync();

When you replace the response body with a custom Stream, this is equivalent to the following:

// Continue processing (additional middleware, controller, etc.)
await next(context);
+ await response.BodyWriter.FlushAsync();

It still might be a good idea. Without doing this, the await replacementResponseBody.CopyToAsync(originalResponseBody) will not copy bytes that have been appended to the BodyWriter via GetMemory()/Advance() without being flushed. AFAIK, gRPC doesn’t flush the BodyWriter for performance reasons when doing normal HTTP/2-based gRPC. It looks like the response does get explicitly flushed when grpc-status: 0 trailer is flushed for grpc-web responses.

I think gRPC should be calling response.CompleteAsync() or response.BodyWriter.CompleteAsync() before exiting its middleware even in the normal non-grpc-web case. This would get the same performance benefit has not flushing for non-wrapped response Streams, but it will flush the stream (without actually completing the real response) when the response Stream is wrapped. @JamesNK

Also, setting body.Position = 0; after ReadToEndAsync() is unnecessary and probably wrong, but it’s not very likely to matter so early in the middleware pipeline.

Possibly related: https://github.com/dotnet/aspnetcore/issues/39317

[halter73]

I think this should probably be fixed in gRPC. If the gRPC logic independently tracked if it has written to the body instead of using response.HasStarted, the simpler MemoryStream response body replacement could still work like it does for most other kinds of requests.

The above is why this issue is currenlty left open in the backlog in case that’s not clear.