Cannot reach grpc service with TLS terminated at edge stack

Describe the bug We’re unable to reach a gRPC service when terminating TLS at edge stack. We’re using ACME to generate certificates and are able to reach http services (e.g. https://py.cloud1.landau.us/). We are able to reach gRPC service without tsl (see repro below).

We have another cluster with same service that terminates TLS at AWS NLB and that one works fine.

To Reproduce Steps to reproduce the behavior: Without TLS:

> evans -r cli list --host mc.cloud1.landau.us --port 443
api.CliService
grpc.reflection.v1alpha.ServerReflection

With TLS:

> evans -r cli list --host mc.cloud1.landau.us --port 443 --tls
evans: failed to run CLI mode: 1 error occurred:

* failed to instantiate the spec: failed to list packages by gRPC reflection: failed to list services from reflection enabled gRPC server: rpc error: code = Unimplemented desc =

Expected behavior I expect gRPC to work with TLS.

Versions (please complete the following information):

• Ambassador: aes:2.2.2 (Installed with Helm)
• Kubernetes environment: AWS EKS
• Version: 1.22

Additional context

Host

apiVersion: getambassador.io/v2
kind: Host
metadata:
  annotations:
    meta.helm.sh/release-name: mission-control
    meta.helm.sh/release-namespace: mike-jetpack-io
  creationTimestamp: "2022-06-01T21:51:33Z"
  generation: 6
  labels:
    app.kubernetes.io/managed-by: Helm
  name: mission-control-app
  namespace: mike-jetpack-io
  resourceVersion: "4203303"
  uid: 6e9be72a-2def-4b8c-9919-90abf8c7b180
spec:
  acmeProvider:
    authority: https://acme-v02.api.letsencrypt.org/directory
    email: [e-mail]
    privateKeySecret:
      name: https-3a-2f-2facme-2dv02.api.letsencrypt.org-2fdirectory--hello-40jetpack.io
    registration: '{"body":{"status":"valid","contact":["mailto:[e-mail]"]},"uri":"https://acme-v02.api.letsencrypt.org/acme/acct/568314936"}'
  ambassador_id:
  - --apiVersion-v3alpha1-only--default
  hostname: mc.cloud1.landau.us
  tls:
    alpn_protocols: h2,http/1.1
    min_tls_version: v1.2
  tlsSecret:
    name: mc.cloud1.landau.us
status:
  state: Ready
  tlsCertificateSource: ACME

Mapping

apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  annotations:
    meta.helm.sh/release-name: mission-control
    meta.helm.sh/release-namespace: mike-jetpack-io
  creationTimestamp: "2022-06-01T22:18:20Z"
  generation: 33
  labels:
    app.kubernetes.io/instance: mission-control
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: app
    app.kubernetes.io/version: latest
    helm.sh/chart: app-0.0.1
  name: mission-control-app
  namespace: mike-jetpack-io
  resourceVersion: "4204105"
  uid: c58aac3b-abb9-4892-af8e-e0bcf06ab4d2
spec:
  ambassador_id:
  - --apiVersion-v3alpha1-only--default
  grpc: true
  host: mc.cloud1.landau.us:443
  prefix: /
  service: mission-control-app.mike-jetpack-io
  timeout_ms: 900000
  tls: upstream # I tried with an without this

Listeners:

> kubectl get listener -n ambassador
NAME                       PORT   PROTOCOL   STACK                    STATSPREFIX   SECURITY   L7DEPTH
edge-stack-listener-8080   8080              ["PROXY","HTTP"]                       XFP
https-listener             8443              ["PROXY","TLS","HTTP"]                 XFP

Things I’ve already tried:

• Created TLSContext (instead of inligning it to host). I tried with both, and each one individual
• I tried adding TCP to listener stack and also changing security
• I tried making the mapping a wildcard, using regex for port, etc
• If I remove grpc: true from mapping, remove port from host, and point it to a http service it works fine

[mikeland86]

Update 🎉

I was able to get it to work by adding strip_matching_host_port: true to the module and removing the port from the mapping. (The docs say either approach works, and when we terminate TLS at the NLB the mapping approach seems to work)

Not sure why, but now not TLS doesn’t work (but that’s fine, we only cared about TLS).

Unsure if related by I also found https://github.com/envoyproxy/envoy/issues/8305

Going to close. Thanks for help on Zoom!

[mikeland86]

Non tls:

[2022-06-02 19:59:35.177][45][debug][router] [source/common/router/router.cc:425] [C278009][S4198637205083794389] cluster 'cluster_mission_control_app_mike_jetpack_io_mike_jetpack_io' match for URL '/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo'

tls:

[2022-06-02 20:00:33.319][45][debug][router] [source/common/router/router.cc:359] [C278104][S15325789635925384087] no cluster match for URL '/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo'

And Envoy replies: (with a 200?)

[2022-06-02 20:00:33.365][45][debug][http] [source/common/http/filter_manager.cc:873] [C278104][S12538254725469138294] Sending local reply with details route_not_found
[2022-06-02 20:00:33.365][45][debug][http] [source/common/http/conn_manager_impl.cc:1501] [C278104][S12538254725469138294] encoding headers via codec (end_stream=true):
':status', '200'
'content-type', 'application/grpc'
'grpc-status', '12'
'date', 'Thu, 02 Jun 2022 20:00:33 GMT'
'server', 'envoy'
--