GRPC :scheme pseudo-header passed from proxy/loadbalancer causes ConnectionAbortedException

Describe the bug

Grpc kestrel .net 5 sample app greeter is running in k8s under http transport, an AWS network load balancer terminates SSL and sends request to the app with :scheme pseudo-header set to “https”, but the actual transport is “http” results in this error - Trace id "0HM6QO8HEK1N9:00000001": HTTP/2 stream error "PROTOCOL_ERROR". A Reset is being sent to the stream. Microsoft.AspNetCore.Connections.ConnectionAbortedException: The request :scheme header 'https' does not match the transport scheme 'http'.

The ask here is to be less stringent on validating on the :schema pseudo header, Go lang sample apps works fine.

To Reproduce

Deploy any sample grpc app from here - https://github.com/grpc/grpc-dotnet/tree/master/examples to k8s, setup the service type loadbalancer using AWS NLB. Test with grpcurl client.

Exceptions (if any)

Microsoft.AspNetCore.Connections.ConnectionAbortedException: The request :scheme header 'https' does not match the transport scheme 'http'.

Further technical details

• ASP.NET Core version
• Include the output of dotnet --info
• The IDE (VS / VS Code/ VS4Mac) you’re running on, and its version

[zhuoyang]

PSA for google cloud run users, google actually fixed this issue on their side

https://issuetracker.google.com/issues/187470251?pli=1

[jimm1419]

This recently broke me in Cloud Run, I guess GCP updated envoy because it was working fine a couple of weeks ago. Here’s a workaround if anyone needs it:

1. Download this repo, I use v5 so git clone https://github.com/dotnet/aspnetcore.git && cd aspnetcore && git checkout tags/v5.0.5
2. git submodule update --init --recursive
3. Find the file /src/Servers/Kestrel/Core/src/Internal/Http2/Http2Stream.cs and comment out lines 244-250. If not version 5.0.5, it’s the block of code that starts with if (!ReferenceEquals(headerScheme, Scheme) &&, comment that whole if block
4. build the project - ./build.sh --configuration Release in the root of the project. It had a few errors but the file I needed was built.
5. Take file aspnetcore/artifacts/bin/Microsoft.AspNetCore.Server.Kestrel.Core/Release/net5.0/Microsoft.AspNetCore.Server.Kestrel.Core.dll and copy it to a folder with the following Dockerfile:

FROM mcr.microsoft.com/dotnet/aspnet:5.0

COPY ./Microsoft.AspNetCore.Server.Kestrel.Core.dll /usr/share/dotnet/shared/Microsoft.AspNetCore.App/5.0.5/

A bit of a pain, but we’re back up on Cloud Run with this …