question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Enabling proxy protocol on AWS NLB makes Emissary to unable to handle requests.

See original GitHub issue

Describe the bug When I enable proxy protocol on my AWS NLB and I configure Emissary Ingress to use it as described in the documentation I am getting following error when sending a request to the cluster:

* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number
* Closing connection 0
curl: (35) error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number

My ambassador Module looks as follows:

apiVersion: getambassador.io/v3alpha1
kind: Module
metadata:
  name: ambassador
  namespace: api-gateway
  labels:
    app.kubernetes.io/name: emissary-ingress
    app.kubernetes.io/part-of: amb
    helm.sh/chart: emissary-ingress-7.1.10
    app.kubernetes.io/instance: amb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: emissary-ingress-ratelimit
    product: aes
spec:
  config:
    defaults:
      httpmapping:
        add_request_headers:
          x-forwarded-proto:
            append: false
            value: https
    diagnostics:
      enabled: false
    envoy_log_type: json
    reject_requests_with_escaped_slashes: true
    server_name: aws-dev-apigateway
    use_proxy_proto: true

My Service looks as follows:

apiVersion: v1
kind: Service
metadata:
  name: amb-emissary-ingress
  namespace: api-gateway
  labels:
    app.kubernetes.io/name: emissary-ingress
    app.kubernetes.io/part-of: amb
    helm.sh/chart: emissary-ingress-7.1.10
    app.kubernetes.io/instance: amb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: ambassador-service
    product: aes
  annotations:
    a8r.io/owner: "Ambassador Labs"
    a8r.io/repository: github.com/datawire/ambassador
    a8r.io/description: "The Ambassador Edge Stack goes beyond traditional API Gateways and Ingress Controllers with the advanced edge features needed to support developer self-service and full-cycle development."
    a8r.io/documentation: https://www.getambassador.io/docs/edge-stack/latest/
    a8r.io/chat: http://a8r.io/Slack
    a8r.io/bugs: https://github.com/datawire/ambassador/issues
    a8r.io/support: https://www.getambassador.io/about-us/support/
    a8r.io/dependencies: amb-emissary-ingress-redis.api-gateway
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-eip-allocations: "eipalloc-xxxxxx"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  type: LoadBalancer
  ports:
    - name: https
      port: 443
      targetPort: 8443
  selector:
    app.kubernetes.io/name: emissary-ingress
    app.kubernetes.io/instance: amb
    profile: main

To Reproduce Steps to reproduce the behaviour:

  1. Configure the Module and the Service as described in the emissary-ingress documentation to properly enable proxy protocol.
  2. Configure AWS NLB and enable Proxy protocol v2
  3. Deploy/Redeploy Emissary-Ingress on the target cluster
  4. See error

Expected behaviour

I am able to send a request, get a response and see an original IP in the upstream service which received the request.

Versions (please complete the following information):

  • Ambassador: emissary-ingress 2.0.5
  • Kubernetes environment: AWS Kubernetes Engine
  • Version: v1.22.2

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:2
  • Comments:6 (3 by maintainers)

github_iconTop GitHub Comments

15reactions
cindymullins-dwcommented, Jan 3, 2022

Adding a working solution, as reported by @tobernguyen (Long Nguyen), a member of our Slack community.

This setup works for me:

apiVersion: getambassador.io/v3alpha1
kind: Listener
metadata:
  name: edge-stack-listener-8080
  namespace: ambassador
spec:
  hostBinding:
    namespace:
      from: ALL
  port: 8080
  protocolStack:
  - PROXY
  - HTTP
  - TCP
  securityModel: INSECURE
---
apiVersion: getambassador.io/v3alpha1
kind: Listener
metadata:
  name: edge-stack-listener-8443
  namespace: ambassador
spec:
  hostBinding:
    namespace:
      from: ALL
  port: 8443
  protocolStack:
  - PROXY
  - TLS
  - HTTP
  - TCP
  securityModel: SECURE
0reactions
ipincecommented, May 31, 2022

Is this the real fix? (in order to avoid using protocolStack and just use HTTPSPROXY?

https://github.com/emissary-ingress/emissary/pull/4154

Read more comments on GitHub >

github_iconTop Results From Across the Web

Configure proxy protocol support for your Classic Load Balancer
Enable proxy protocol using the AWS CLI. To enable proxy protocol, you must create a policy of type ProxyProtocolPolicyType and then enable the...
Read more >
Supporting AWS proxy protocol · Issue #3300 - GitHub
The purpose of this issue is to request a high level configuration option for the chart to deploy a proxy_protocol ready and enabled...
Read more >
Emissary-ingress with AWS | Ambassador
The proxy protocol is a wrapper around an HTTP request that, like X-Forwarded-For , lists the IP address of the downstream connecting client...
Read more >
Trouble enabling Proxy Protocol on AWS ELB - Server Fault
I can make requests successfully through the back-end servers with both HTTP and HTTPS but I just don't seem to get the expected...
Read more >
Recommended settings for AWS layer 4 load balancer(NLB ...
This config creates a classic load balancer, not NLB as stated. · If we enable proxy protocol using the annotation: service.beta.kubernetes.io/ ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found