question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

(aws-eks): Construct Library custom resources doesn't use proxy properly

See original GitHub issue

I’m trying to test new feature related to provision within the VPC lambda functions related to EKS configuration like ClusterHandler. Lambdas are placed to vpc and it’s great. But I got error connecting to EKS api via proxy

Reproduction Steps

self.cluster = aws_eks.Cluster(
            scope=self,
            id='cluster',
            cluster_name="cluster-" + environment,
            endpoint_access=aws_eks.EndpointAccess.PUBLIC_AND_PRIVATE,
            default_capacity=0,
            vpc=vpc,
            vpc_subnets=[aws_ec2.SubnetSelection(subnets=[subnet_a_eks, subnet_b_eks, subnet_c_eks])],
            # issue with 3 subnet
            place_cluster_handler_in_vpc=True,
            version=cluster_version,
            cluster_handler_environment={
                "http_proxy": "http://login:pass@proxy.cloud.local:8080/"
            },
            kubectl_environment={
                "http_proxy": "http://login:pass@proxy.cloud.local:8080/"
            },
            security_group=eks_control_plane_sg,
            role=eks_control_plane_role,
        )

What did you expect to happen?

successful cluster creation

What actually happened?

Cloudwatch log of ProviderframeworkonEvent. function:
2021-01-09T14:37:27.905Z e785fa78-c5f8-471c-a495-59d75389a6c6 INFO [provider-framework] submit response to cloudformation { "Status": "FAILED", "Reason": "Error: connect ETIMEDOUT 63.32.73.253:443\n at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)", "StackId": "arn:aws:cloudformation:eu-west-1:accountid:stack/eks-stack-develop-cdk/276db000-5285-11eb-ab35-0615947f7f49", "RequestId": "ce8a03d7-fdf3-4def-acb4-6219fb352732", "PhysicalResourceId": "AWSCDK::CustomResourceProviderFramework::CREATE_FAILED", "LogicalResourceId": "clusterC5B25D0D" }

Proxy is fine i tested it against same endpoint ( strange that it calls ec2 and nor eks api

[ec2-user@ip-10-60-233-255 ~]$ curl -vk https://ec2-63-32-73-253.eu-west-1.compute.amazonaws.com.
* Rebuilt URL to: https://ec2-63-32-73-253.eu-west-1.compute.amazonaws.com./
* Uses proxy env variable https_proxy == 'http://user:password@proxy.cloud.local:8080/'
*   Trying 10.60.249.170...
* TCP_NODELAY set
* Connected to proxy.cloud.local (10.60.249.170) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to ec2-63-32-73-253.eu-west-1.compute.amazonaws.com:443
* Proxy auth using Basic with user 'user'
> CONNECT ec2-63-32-73-253.eu-west-1.compute.amazonaws.com:443 HTTP/1.1
> Host: ec2-63-32-73-253.eu-west-1.compute.amazonaws.com:443
> Proxy-Authorization: Basic token
> User-Agent: curl/7.61.1
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=lambda.eu-west-1.amazonaws.com
*  start date: Dec 23 00:00:00 2020 GMT
*  expire date: Jan 21 23:59:59 2022 GMT
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x17f8190)
> GET / HTTP/2
> Host: ec2-63-32-73-253.eu-west-1.compute.amazonaws.com
> User-Agent: curl/7.61.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 403 
< date: Tue, 12 Jan 2021 13:31:59 GMT
< content-length: 127
< x-amzn-requestid: c6c6badc-56de-4e6a-8266-0d7971505c84
< 
<MissingAuthenticationTokenException>
  <Message>Missing Authentication Token</Message>
</MissingAuthenticationTokenException>
* Connection #0 to host proxy.cloud.local left intact

Environment

  • **CDK CLI Version :1.83
  • Framework Version:
  • **Node.js Version: v14.13.0
  • **OS :macOS
  • **Language (Version): Python (3.9)

Other

I noticed that 5 lambda functions are created but only 1 of them “OnEventHandler” receives proxy configuration, but looks like it only one which interacts with api


This is 🐛 Bug Report

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:11
  • Comments:12 (5 by maintainers)

github_iconTop GitHub Comments

2reactions
iliapolocommented, Jan 20, 2021

@alexey-boyko

It happens anyway even if i add proxy variable or not

I guess that makes in case the proxy isn’t being applied correctly. I did a little digging and it seems that unlike python, nodejs doesn’t use any global env variables to configures proxy passes. Looks like every http client implements this independently.

We might need to do add explicit support for this in the library. I’ll try adding some more information soon.

0reactions
github-actions[bot]commented, Sep 23, 2021

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

Read more comments on GitHub >

github_iconTop Results From Across the Web

aws-cdk/aws-eks module - AWS Documentation
By default, this library will allocate a managed node group with 2 m5.large instances (this instance type suits most common use-cases, and is...
Read more >
Amazon EKS add-ons: Advanced configuration | Containers
Given an existing Amazon EKS cluster, we want to use the AWS CLI to add the VPC CNI add-on to the existing cluster...
Read more >
aws-cdk/custom-resources module - AWS Documentation
AWS CloudFormation custom resources are extension points to the provisioning engine. When CloudFormation needs to create, update or delete a custom resource ......
Read more >
Amazon EKS troubleshooting - AWS Documentation
This chapter covers some common errors that you may see while using Amazon EKS and how to work around them. If you need...
Read more >
Amazon EKS Construct Library - AWS Documentation
At the moment, the AWS::EKS::Cluster AWS CloudFormation resource does not support this behavior, so in order to support “programmatic kubectl”, such as applying ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found