Cannot reach grpc service with TLS terminated at edge stack
See original GitHub issueDescribe the bug
We’re unable to reach a gRPC service when terminating TLS at edge stack. We’re using ACME to generate certificates and are able to reach http services (e.g. https://py.cloud1.landau.us/). We are able to reach gRPC service without tsl
(see repro below).
We have another cluster with same service that terminates TLS at AWS NLB and that one works fine.
To Reproduce Steps to reproduce the behavior: Without TLS:
> evans -r cli list --host mc.cloud1.landau.us --port 443
api.CliService
grpc.reflection.v1alpha.ServerReflection
With TLS:
> evans -r cli list --host mc.cloud1.landau.us --port 443 --tls
evans: failed to run CLI mode: 1 error occurred:
* failed to instantiate the spec: failed to list packages by gRPC reflection: failed to list services from reflection enabled gRPC server: rpc error: code = Unimplemented desc =
Expected behavior I expect gRPC to work with TLS.
Versions (please complete the following information):
- Ambassador: aes:2.2.2 (Installed with Helm)
- Kubernetes environment: AWS EKS
- Version: 1.22
Additional context
Host
apiVersion: getambassador.io/v2
kind: Host
metadata:
annotations:
meta.helm.sh/release-name: mission-control
meta.helm.sh/release-namespace: mike-jetpack-io
creationTimestamp: "2022-06-01T21:51:33Z"
generation: 6
labels:
app.kubernetes.io/managed-by: Helm
name: mission-control-app
namespace: mike-jetpack-io
resourceVersion: "4203303"
uid: 6e9be72a-2def-4b8c-9919-90abf8c7b180
spec:
acmeProvider:
authority: https://acme-v02.api.letsencrypt.org/directory
email: hello@jetpack.io
privateKeySecret:
name: https-3a-2f-2facme-2dv02.api.letsencrypt.org-2fdirectory--hello-40jetpack.io
registration: '{"body":{"status":"valid","contact":["mailto:hello@jetpack.io"]},"uri":"https://acme-v02.api.letsencrypt.org/acme/acct/568314936"}'
ambassador_id:
- --apiVersion-v3alpha1-only--default
hostname: mc.cloud1.landau.us
tls:
alpn_protocols: h2,http/1.1
min_tls_version: v1.2
tlsSecret:
name: mc.cloud1.landau.us
status:
state: Ready
tlsCertificateSource: ACME
Mapping
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
annotations:
meta.helm.sh/release-name: mission-control
meta.helm.sh/release-namespace: mike-jetpack-io
creationTimestamp: "2022-06-01T22:18:20Z"
generation: 33
labels:
app.kubernetes.io/instance: mission-control
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: app
app.kubernetes.io/version: latest
helm.sh/chart: app-0.0.1
name: mission-control-app
namespace: mike-jetpack-io
resourceVersion: "4204105"
uid: c58aac3b-abb9-4892-af8e-e0bcf06ab4d2
spec:
ambassador_id:
- --apiVersion-v3alpha1-only--default
grpc: true
host: mc.cloud1.landau.us:443
prefix: /
service: mission-control-app.mike-jetpack-io
timeout_ms: 900000
tls: upstream # I tried with an without this
Listeners:
> kubectl get listener -n ambassador
NAME PORT PROTOCOL STACK STATSPREFIX SECURITY L7DEPTH
edge-stack-listener-8080 8080 ["PROXY","HTTP"] XFP
https-listener 8443 ["PROXY","TLS","HTTP"] XFP
Things I’ve already tried:
- Created TLSContext (instead of inligning it to host). I tried with both, and each one individual
- I tried adding TCP to listener stack and also changing security
- I tried making the mapping a wildcard, using regex for port, etc
- If I remove
grpc: true
from mapping, remove port fromhost
, and point it to a http service it works fine
Issue Analytics
- State:
- Created a year ago
- Comments:5 (1 by maintainers)
Top Results From Across the Web
Error Accessing C#-based gRPC Server when hosted behind ...
Add a TLS context based on an existing secret according to the tutorial; Try to connect via a C# gRPC Client via a...
Read more >gRPC Connections | Edge Stack
Since Ambassador Edge Stack terminates TLS by default, in the Host we add a requestPolicy which allows insecure connections. After adding the Ambassador...
Read more >Terminate grpc traffic with nginx - Stack Overflow
This recent blogpost says that nginx is able to terminate http/2 and grpc traffic. From all the illustrations and the text, it does...
Read more >Troubleshoot gRPC on .NET Core - Microsoft Learn
The gRPC template and samples use Transport Layer Security (TLS) to secure gRPC services by default. gRPC clients need to use a secure ......
Read more >The Mysterious Gotcha of gRPC Stream Performance | Ably Blog
gRPC is highly useful for fast, efficient data exchange and client/server state sync. Here's a performance gotcha we ran across.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Update 🎉
I was able to get it to work by adding
strip_matching_host_port: true
to the module and removing the port from the mapping. (The docs say either approach works, and when we terminate TLS at the NLB the mapping approach seems to work)Not sure why, but now not TLS doesn’t work (but that’s fine, we only cared about TLS).
Unsure if related by I also found https://github.com/envoyproxy/envoy/issues/8305
Going to close. Thanks for help on Zoom!
Non tls:
[2022-06-02 19:59:35.177][45][debug][router] [source/common/router/router.cc:425] [C278009][S4198637205083794389] cluster 'cluster_mission_control_app_mike_jetpack_io_mike_jetpack_io' match for URL '/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo'
tls:
[2022-06-02 20:00:33.319][45][debug][router] [source/common/router/router.cc:359] [C278104][S15325789635925384087] no cluster match for URL '/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo'
And Envoy replies: (with a 200?)