Enabling proxy protocol on AWS NLB makes Emissary to unable to handle requests.
See original GitHub issueDescribe the bug When I enable proxy protocol on my AWS NLB and I configure Emissary Ingress to use it as described in the documentation I am getting following error when sending a request to the cluster:
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
* CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number
* Closing connection 0
curl: (35) error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number
My ambassador Module
looks as follows:
apiVersion: getambassador.io/v3alpha1
kind: Module
metadata:
name: ambassador
namespace: api-gateway
labels:
app.kubernetes.io/name: emissary-ingress
app.kubernetes.io/part-of: amb
helm.sh/chart: emissary-ingress-7.1.10
app.kubernetes.io/instance: amb
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: emissary-ingress-ratelimit
product: aes
spec:
config:
defaults:
httpmapping:
add_request_headers:
x-forwarded-proto:
append: false
value: https
diagnostics:
enabled: false
envoy_log_type: json
reject_requests_with_escaped_slashes: true
server_name: aws-dev-apigateway
use_proxy_proto: true
My Service
looks as follows:
apiVersion: v1
kind: Service
metadata:
name: amb-emissary-ingress
namespace: api-gateway
labels:
app.kubernetes.io/name: emissary-ingress
app.kubernetes.io/part-of: amb
helm.sh/chart: emissary-ingress-7.1.10
app.kubernetes.io/instance: amb
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: ambassador-service
product: aes
annotations:
a8r.io/owner: "Ambassador Labs"
a8r.io/repository: github.com/datawire/ambassador
a8r.io/description: "The Ambassador Edge Stack goes beyond traditional API Gateways and Ingress Controllers with the advanced edge features needed to support developer self-service and full-cycle development."
a8r.io/documentation: https://www.getambassador.io/docs/edge-stack/latest/
a8r.io/chat: http://a8r.io/Slack
a8r.io/bugs: https://github.com/datawire/ambassador/issues
a8r.io/support: https://www.getambassador.io/about-us/support/
a8r.io/dependencies: amb-emissary-ingress-redis.api-gateway
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-eip-allocations: "eipalloc-xxxxxx"
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
type: LoadBalancer
ports:
- name: https
port: 443
targetPort: 8443
selector:
app.kubernetes.io/name: emissary-ingress
app.kubernetes.io/instance: amb
profile: main
To Reproduce Steps to reproduce the behaviour:
- Configure the
Module
and theService
as described in the emissary-ingress documentation to properly enableproxy
protocol. - Configure AWS NLB and enable Proxy protocol v2
- Deploy/Redeploy Emissary-Ingress on the target cluster
- See error
Expected behaviour
I am able to send a request, get a response and see an original IP in the upstream service which received the request.
Versions (please complete the following information):
- Ambassador: emissary-ingress 2.0.5
- Kubernetes environment: AWS Kubernetes Engine
- Version: v1.22.2
Issue Analytics
- State:
- Created 2 years ago
- Reactions:2
- Comments:6 (3 by maintainers)
Top Results From Across the Web
Configure proxy protocol support for your Classic Load Balancer
Enable proxy protocol using the AWS CLI. To enable proxy protocol, you must create a policy of type ProxyProtocolPolicyType and then enable the...
Read more >Supporting AWS proxy protocol · Issue #3300 - GitHub
The purpose of this issue is to request a high level configuration option for the chart to deploy a proxy_protocol ready and enabled...
Read more >Emissary-ingress with AWS | Ambassador
The proxy protocol is a wrapper around an HTTP request that, like X-Forwarded-For , lists the IP address of the downstream connecting client...
Read more >Trouble enabling Proxy Protocol on AWS ELB - Server Fault
I can make requests successfully through the back-end servers with both HTTP and HTTPS but I just don't seem to get the expected...
Read more >Recommended settings for AWS layer 4 load balancer(NLB ...
This config creates a classic load balancer, not NLB as stated. · If we enable proxy protocol using the annotation: service.beta.kubernetes.io/ ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Adding a working solution, as reported by @tobernguyen (Long Nguyen), a member of our Slack community.
This setup works for me:
Is this the real fix? (in order to avoid using protocolStack and just use
HTTPSPROXY
?https://github.com/emissary-ingress/emissary/pull/4154