question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Cannot reach grpc service with TLS terminated at edge stack

See original GitHub issue

Describe the bug We’re unable to reach a gRPC service when terminating TLS at edge stack. We’re using ACME to generate certificates and are able to reach http services (e.g. https://py.cloud1.landau.us/). We are able to reach gRPC service without tsl (see repro below).

We have another cluster with same service that terminates TLS at AWS NLB and that one works fine.

To Reproduce Steps to reproduce the behavior: Without TLS:

> evans -r cli list --host mc.cloud1.landau.us --port 443
api.CliService
grpc.reflection.v1alpha.ServerReflection

With TLS:

> evans -r cli list --host mc.cloud1.landau.us --port 443 --tls
evans: failed to run CLI mode: 1 error occurred:

* failed to instantiate the spec: failed to list packages by gRPC reflection: failed to list services from reflection enabled gRPC server: rpc error: code = Unimplemented desc =

Expected behavior I expect gRPC to work with TLS.

Versions (please complete the following information):

  • Ambassador: aes:2.2.2 (Installed with Helm)
  • Kubernetes environment: AWS EKS
  • Version: 1.22

Additional context

Host

apiVersion: getambassador.io/v2
kind: Host
metadata:
  annotations:
    meta.helm.sh/release-name: mission-control
    meta.helm.sh/release-namespace: mike-jetpack-io
  creationTimestamp: "2022-06-01T21:51:33Z"
  generation: 6
  labels:
    app.kubernetes.io/managed-by: Helm
  name: mission-control-app
  namespace: mike-jetpack-io
  resourceVersion: "4203303"
  uid: 6e9be72a-2def-4b8c-9919-90abf8c7b180
spec:
  acmeProvider:
    authority: https://acme-v02.api.letsencrypt.org/directory
    email: hello@jetpack.io
    privateKeySecret:
      name: https-3a-2f-2facme-2dv02.api.letsencrypt.org-2fdirectory--hello-40jetpack.io
    registration: '{"body":{"status":"valid","contact":["mailto:hello@jetpack.io"]},"uri":"https://acme-v02.api.letsencrypt.org/acme/acct/568314936"}'
  ambassador_id:
  - --apiVersion-v3alpha1-only--default
  hostname: mc.cloud1.landau.us
  tls:
    alpn_protocols: h2,http/1.1
    min_tls_version: v1.2
  tlsSecret:
    name: mc.cloud1.landau.us
status:
  state: Ready
  tlsCertificateSource: ACME

Mapping

apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  annotations:
    meta.helm.sh/release-name: mission-control
    meta.helm.sh/release-namespace: mike-jetpack-io
  creationTimestamp: "2022-06-01T22:18:20Z"
  generation: 33
  labels:
    app.kubernetes.io/instance: mission-control
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: app
    app.kubernetes.io/version: latest
    helm.sh/chart: app-0.0.1
  name: mission-control-app
  namespace: mike-jetpack-io
  resourceVersion: "4204105"
  uid: c58aac3b-abb9-4892-af8e-e0bcf06ab4d2
spec:
  ambassador_id:
  - --apiVersion-v3alpha1-only--default
  grpc: true
  host: mc.cloud1.landau.us:443
  prefix: /
  service: mission-control-app.mike-jetpack-io
  timeout_ms: 900000
  tls: upstream # I tried with an without this

Listeners:

> kubectl get listener -n ambassador
NAME                       PORT   PROTOCOL   STACK                    STATSPREFIX   SECURITY   L7DEPTH
edge-stack-listener-8080   8080              ["PROXY","HTTP"]                       XFP
https-listener             8443              ["PROXY","TLS","HTTP"]                 XFP

Things I’ve already tried:

  • Created TLSContext (instead of inligning it to host). I tried with both, and each one individual
  • I tried adding TCP to listener stack and also changing security
  • I tried making the mapping a wildcard, using regex for port, etc
  • If I remove grpc: true from mapping, remove port from host, and point it to a http service it works fine

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:5 (1 by maintainers)

github_iconTop GitHub Comments

1reaction
mikeland86commented, Jun 2, 2022

Update 🎉

I was able to get it to work by adding strip_matching_host_port: true to the module and removing the port from the mapping. (The docs say either approach works, and when we terminate TLS at the NLB the mapping approach seems to work)

Not sure why, but now not TLS doesn’t work (but that’s fine, we only cared about TLS).

Unsure if related by I also found https://github.com/envoyproxy/envoy/issues/8305

Going to close. Thanks for help on Zoom!

0reactions
mikeland86commented, Jun 2, 2022

Non tls:

[2022-06-02 19:59:35.177][45][debug][router] [source/common/router/router.cc:425] [C278009][S4198637205083794389] cluster 'cluster_mission_control_app_mike_jetpack_io_mike_jetpack_io' match for URL '/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo'

tls:

[2022-06-02 20:00:33.319][45][debug][router] [source/common/router/router.cc:359] [C278104][S15325789635925384087] no cluster match for URL '/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo'

And Envoy replies: (with a 200?)

[2022-06-02 20:00:33.365][45][debug][http] [source/common/http/filter_manager.cc:873] [C278104][S12538254725469138294] Sending local reply with details route_not_found
[2022-06-02 20:00:33.365][45][debug][http] [source/common/http/conn_manager_impl.cc:1501] [C278104][S12538254725469138294] encoding headers via codec (end_stream=true):
':status', '200'
'content-type', 'application/grpc'
'grpc-status', '12'
'date', 'Thu, 02 Jun 2022 20:00:33 GMT'
'server', 'envoy'
--
Read more comments on GitHub >

github_iconTop Results From Across the Web

Error Accessing C#-based gRPC Server when hosted behind ...
Add a TLS context based on an existing secret according to the tutorial; Try to connect via a C# gRPC Client via a...
Read more >
gRPC Connections | Edge Stack
Since Ambassador Edge Stack terminates TLS by default, in the Host we add a requestPolicy which allows insecure connections. After adding the Ambassador...
Read more >
Terminate grpc traffic with nginx - Stack Overflow
This recent blogpost says that nginx is able to terminate http/2 and grpc traffic. From all the illustrations and the text, it does...
Read more >
Troubleshoot gRPC on .NET Core - Microsoft Learn
The gRPC template and samples use Transport Layer Security (TLS) to secure gRPC services by default. gRPC clients need to use a secure ......
Read more >
The Mysterious Gotcha of gRPC Stream Performance | Ably Blog
gRPC is highly useful for fast, efficient data exchange and client/server state sync. Here's a performance gotcha we ran across.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found