Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

(aws-eks): create failed [ Cluster/AwsAuth/manifest/Resource/Default (ClusterAwsAuthmanifestFE51F8AE)]

See original GitHub issue

import ec2 = require('@aws-cdk/aws-ec2');
import eks = require('@aws-cdk/aws-eks');
import cdk = require('@aws-cdk/core');
//import iam = require('@aws-cdk/aws-iam');

class EKSCluster extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const vpcId = "vpc-xxxxxx";
    const getExistingVpc = ec2.Vpc.fromLookup(this, 'xxxx-vpc',{isDefault: false,vpcId: vpcId });

    const projectName = "xxxxx";
    const environment = "alpha";
    const clusterName = projectName+"-"+environment+"-eks";

    const clusterSecurityGroup = new ec2.SecurityGroup(this,"eks-sg",{
      vpc: getExistingVpc,
      securityGroupName: clusterName+"-sg",
      allowAllOutbound: true,
      description: 'CDK eks Security Group'
    });
    clusterSecurityGroup.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(80));
    clusterSecurityGroup.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(443));
    clusterSecurityGroup.addIngressRule(ec2.Peer.ipv4("10.151.0.0/16"), ec2.Port.tcpRange(1,65000))


    const eksCluster = new eks.Cluster(this, 'Cluster', {
      vpc: getExistingVpc,
      version: eks.KubernetesVersion.V1_18,
      clusterName: clusterName,
      defaultCapacity: 0,
      vpcSubnets: [{ subnetType: ec2.SubnetType.PRIVATE }],
      securityGroup: clusterSecurityGroup
    });
    eksCluster.addNodegroupCapacity('www-node', {
      nodegroupName: "www",
      instanceTypes: [new ec2.InstanceType('t3.xlarge')],
      subnets: {
        subnetType: ec2.SubnetType.PRIVATE,
        availabilityZones: ['ap-northeast-2a', 'ap-northeast-2c']
      },
      minSize: 2,
      desiredSize: 2,
      maxSize: 2,
      diskSize: 20,
      amiType: eks.NodegroupAmiType.AL2_X86_64,
      labels: {"app": "test" },
      tags: {"Project": projectName}
    });

  }
}

const app = new cdk.App();
new EKSCluster(app, 'xxxxxxxx',{
  env : {
    account: process.env.CDK_DEFAULT_ACCOUNT,
    region: process.env.CDK_DEFAULT_REGION,
  },
});

app.synth();

cdk create very very long almost 1hour time out failed
result: eks master network subnet public+private (i want only private subnet)

vpcSubnets: [{ subnetType: ec2.SubnetType.PRIVATE }],

log

multiTestClusterA: creating CloudFormation changeset...
  0/14 | 오후 6:33:27 | REVIEW_IN_PROGRESS   | AWS::CloudFormation::Stack            | multiTestClusterA User Initiated
  0/14 | 오후 6:33:32 | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack            | multiTestClusterA User Initiated
  0/14 | 오후 6:34:00 | CREATE_IN_PROGRESS   | AWS::IAM::Role                        | Cluster/Resource/CreationRole (ClusterCreationRole360249B6)
  0/14 | 오후 6:34:00 | CREATE_IN_PROGRESS   | AWS::IAM::Role                        | Cluster/Role (ClusterRoleFA261979)
  0/14 | 오후 6:34:00 | CREATE_IN_PROGRESS   | AWS::IAM::Role                        | Cluster/Nodegroupwww-node/NodeGroupRole (ClusterNodegroupwwwnodeNodeGroupRole9ADD7437)
  0/14 | 오후 6:34:00 | CREATE_IN_PROGRESS   | AWS::CDK::Metadata                    | @aws-cdk--aws-eks.ClusterResourceProvider/CDKMetadata/Default (CDKMetadata)
  0/14 | 오후 6:34:01 | CREATE_IN_PROGRESS   | AWS::IAM::Role                        | Cluster/Resource/CreationRole (ClusterCreationRole360249B6) Resource creation Initiated
  0/14 | 오후 6:34:01 | CREATE_IN_PROGRESS   | AWS::IAM::Role                        | Cluster/Role (ClusterRoleFA261979) Resource creation Initiated
  0/14 | 오후 6:34:01 | CREATE_IN_PROGRESS   | AWS::IAM::Role                        | Cluster/MastersRole (ClusterMastersRole9AA35625)
  0/14 | 오후 6:34:01 | CREATE_IN_PROGRESS   | AWS::IAM::Role                        | Cluster/Nodegroupwww-node/NodeGroupRole (ClusterNodegroupwwwnodeNodeGroupRole9ADD7437) Resource creation Initiated
  0/14 | 오후 6:34:01 | CREATE_IN_PROGRESS   | AWS::EC2::SecurityGroup               | eks-sg (ekssgD0B8BE47)
  0/14 | 오후 6:34:01 | CREATE_IN_PROGRESS   | AWS::IAM::Role                        | Cluster/MastersRole (ClusterMastersRole9AA35625) Resource creation Initiated
  0/14 | 오후 6:34:02 | CREATE_IN_PROGRESS   | AWS::CDK::Metadata                    | @aws-cdk--aws-eks.ClusterResourceProvider/CDKMetadata/Default (CDKMetadata) Resource creation Initiated
  1/14 | 오후 6:34:03 | CREATE_COMPLETE      | AWS::CDK::Metadata                    | @aws-cdk--aws-eks.ClusterResourceProvider/CDKMetadata/Default (CDKMetadata)
  1/14 | 오후 6:34:06 | CREATE_IN_PROGRESS   | AWS::EC2::SecurityGroup               | eks-sg (ekssgD0B8BE47) Resource creation Initiated
  2/14 | 오후 6:34:08 | CREATE_COMPLETE      | AWS::EC2::SecurityGroup               | eks-sg (ekssgD0B8BE47)
  6/14 | 오후 6:34:21 | CREATE_COMPLETE      | AWS::IAM::Role                        | Cluster/Resource/CreationRole (ClusterCreationRole360249B6)
  6/14 | 오후 6:34:21 | CREATE_COMPLETE      | AWS::IAM::Role                        | Cluster/Role (ClusterRoleFA261979)
  6/14 | 오후 6:34:21 | CREATE_COMPLETE      | AWS::IAM::Role                        | Cluster/MastersRole (ClusterMastersRole9AA35625)
  6/14 | 오후 6:34:22 | CREATE_COMPLETE      | AWS::IAM::Role                        | Cluster/Nodegroupwww-node/NodeGroupRole (ClusterNodegroupwwwnodeNodeGroupRole9ADD7437)
  6/14 | 오후 6:34:24 | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack            | @aws-cdk--aws-eks.ClusterResourceProvider.NestedStack/@aws-cdk--aws-eks.ClusterResourceProvider.NestedStackResource (awscdkawseksClusterResourceProviderNestedStac
kawscdkawseksClusterResourceProviderNestedStackResource9827C454)
  6/14 | 오후 6:34:25 | CREATE_IN_PROGRESS   | AWS::IAM::Policy                      | Cluster/Resource/CreationRole/DefaultPolicy (ClusterCreationRoleDefaultPolicyE8BDFC7B)
  6/14 | 오후 6:34:25 | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack            | @aws-cdk--aws-eks.ClusterResourceProvider.NestedStack/@aws-cdk--aws-eks.ClusterResourceProvider.NestedStackResource (awscdkawseksClusterResourceProviderNestedStac
kawscdkawseksClusterResourceProviderNestedStackResource9827C454) Resource creation Initiated
  6/14 | 오후 6:34:26 | CREATE_IN_PROGRESS   | AWS::IAM::Policy                      | Cluster/Resource/CreationRole/DefaultPolicy (ClusterCreationRoleDefaultPolicyE8BDFC7B) Resource creation Initiated
  7/14 | 오후 6:34:47 | CREATE_COMPLETE      | AWS::IAM::Policy                      | Cluster/Resource/CreationRole/DefaultPolicy (ClusterCreationRoleDefaultPolicyE8BDFC7B)
 7/14 Currently in progress: multiTestClusterA, awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourceProviderNestedStackResource9827C454
  8/14 | 오후 6:37:29 | CREATE_COMPLETE      | AWS::CloudFormation::Stack            | @aws-cdk--aws-eks.ClusterResourceProvider.NestedStack/@aws-cdk--aws-eks.ClusterResourceProvider.NestedStackResource (awscdkawseksClusterResourceProviderNestedStac
kawscdkawseksClusterResourceProviderNestedStackResource9827C454)
  8/14 | 오후 6:37:33 | CREATE_IN_PROGRESS   | Custom::AWSCDK-EKS-Cluster            | Cluster/Resource/Resource/Default (Cluster9EE0221C)
 8/14 Currently in progress: multiTestClusterA, Cluster9EE0221C
  9/14 | 오후 6:50:07 | CREATE_IN_PROGRESS   | Custom::AWSCDK-EKS-Cluster            | Cluster/Resource/Resource/Default (Cluster9EE0221C) Resource creation Initiated
  9/14 | 오후 6:50:08 | CREATE_COMPLETE      | Custom::AWSCDK-EKS-Cluster            | Cluster/Resource/Resource/Default (Cluster9EE0221C)
 10/14 | 오후 6:50:12 | CREATE_IN_PROGRESS   | AWS::SSM::Parameter                   | Cluster/KubectlReadyBarrier (ClusterKubectlReadyBarrier200052AF)
 10/14 | 오후 6:50:12 | CREATE_IN_PROGRESS   | AWS::EKS::Nodegroup                   | Cluster/Nodegroupwww-node (ClusterNodegroupwwwnode4C558510)
 10/14 | 오후 6:50:12 | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack            | @aws-cdk--aws-eks.KubectlProvider.NestedStack/@aws-cdk--aws-eks.KubectlProvider.NestedStackResource (awscdkawseksKubectlProviderNestedStackawscdkawseksKubectlProv
iderNestedStackResourceA7AEBA6B)
 10/14 | 오후 6:50:13 | CREATE_IN_PROGRESS   | AWS::SSM::Parameter                   | Cluster/KubectlReadyBarrier (ClusterKubectlReadyBarrier200052AF) Resource creation Initiated
 10/14 | 오후 6:50:13 | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack            | @aws-cdk--aws-eks.KubectlProvider.NestedStack/@aws-cdk--aws-eks.KubectlProvider.NestedStackResource (awscdkawseksKubectlProviderNestedStackawscdkawseksKubectlProv
iderNestedStackResourceA7AEBA6B) Resource creation Initiated
 10/14 | 오후 6:50:14 | CREATE_COMPLETE      | AWS::SSM::Parameter                   | Cluster/KubectlReadyBarrier (ClusterKubectlReadyBarrier200052AF)
 10/14 | 오후 6:50:25 | CREATE_IN_PROGRESS   | AWS::EKS::Nodegroup                   | Cluster/Nodegroupwww-node (ClusterNodegroupwwwnode4C558510) Resource creation Initiated
10/14 Currently in progress: multiTestClusterA, ClusterNodegroupwwwnode4C558510, awscdkawseksKubectlProviderNestedStackawscdkawseksKubectlProviderNestedStackResourceA7AEBA6B
 11/14 | 오후 6:52:54 | CREATE_COMPLETE      | AWS::EKS::Nodegroup                   | Cluster/Nodegroupwww-node (ClusterNodegroupwwwnode4C558510)
11/14 Currently in progress: multiTestClusterA, awscdkawseksKubectlProviderNestedStackawscdkawseksKubectlProviderNestedStackResourceA7AEBA6B
 12/14 | 오후 6:56:53 | CREATE_COMPLETE      | AWS::CloudFormation::Stack            | @aws-cdk--aws-eks.KubectlProvider.NestedStack/@aws-cdk--aws-eks.KubectlProvider.NestedStackResource (awscdkawseksKubectlProviderNestedStackawscdkawseksKubectlProv
iderNestedStackResourceA7AEBA6B)
 12/14 | 오후 6:56:58 | CREATE_IN_PROGRESS   | Custom::AWSCDK-EKS-KubernetesResource | Cluster/AwsAuth/manifest/Resource/Default (ClusterAwsAuthmanifestFE51F8AE)
12/14 Currently in progress: multiTestClusterA, ClusterAwsAuthmanifestFE51F8AE
 12/14 | 오후 7:57:14 | CREATE_FAILED        | Custom::AWSCDK-EKS-KubernetesResource | Cluster/AwsAuth/manifest/Resource/Default (ClusterAwsAuthmanifestFE51F8AE) Custom Resource failed to stabilize in expected time. If you are using the Python cfn-r
esponse module, you may need to update your Lambda function code so that CloudFormation can attach the updated version.
        new CustomResource (E:\project\cloud-formation\cdk\eks-sample\cluster\node_modules\@aws-cdk\core\lib\custom-resource.js:28:25)
        \_ new KubernetesManifest (E:\project\cloud-formation\cdk\eks-sample\cluster\node_modules\@aws-cdk\aws-eks\lib\k8s-manifest.js:34:9)
        \_ new AwsAuth (E:\project\cloud-formation\cdk\eks-sample\cluster\node_modules\@aws-cdk\aws-eks\lib\aws-auth.js:25:9)
        \_ Cluster.get awsAuth [as awsAuth] (E:\project\cloud-formation\cdk\eks-sample\cluster\node_modules\@aws-cdk\aws-eks\lib\cluster.js:564:29)
        \_ new Cluster (E:\project\cloud-formation\cdk\eks-sample\cluster\node_modules\@aws-cdk\aws-eks\lib\cluster.js:359:14)
        \_ new EKSCluster (E:\project\cloud-formation\cdk\eks-sample\cluster\index.js:45:28)
        \_ Object.<anonymous> (E:\project\cloud-formation\cdk\eks-sample\cluster\index.js:81:1)
        \_ Module._compile (internal/modules/cjs/loader.js:1063:30)
        \_ Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
        \_ Module.load (internal/modules/cjs/loader.js:928:32)
        \_ Function.Module._load (internal/modules/cjs/loader.js:769:14)
        \_ Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
        \_ internal/main/run_main_module.js:17:47
 12/14 | 오후 7:57:17 | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack            | multiTestClusterA The following resource(s) failed to create: [ClusterAwsAuthmanifestFE51F8AE]. Rollback requested by user.

Reproduction Steps

What did you expect to happen?

What actually happened?

Environment

npm: ‘6.14.8’, node: v14.15.1 cdk 1.89.0 (build df7253c)

Other

This is 🐛 Bug Report

Issue Analytics

State:
Created 3 years ago
Comments:9 (3 by maintainers)

Top GitHub Comments

1reaction

robythomascommented, Sep 2, 2022

I’ve asked a related question in StackOverflow for which I don’t have any response in it yet.

It fails typically at

MyStack | 10/12 | 4:00:48 pm | CREATE_FAILED        | Custom::AWSCDK-EKS-KubernetesResource | 
Cluster/AwsAuth/manifest/Resource/Default (ClusterAwsAuthmanifestFE51F8AE) 
Received response status [FAILED] from custom resource. 
Message returned: Error: connect ETIMEDOUT 34.223.18.133:443
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1159:16) (RequestId: bf13e37a-c6ec-432c-9147-cfc6f6ba8007)

In my case, increasing the defaultCapacity does not help as indicated by @maieve. Checked the VPC endpoint policy as suggested by @cjcooper and I see it’s

{
	"Version": "2008-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Principal": "*",
			"Action": "*",
			"Resource": "*"
		}
	]
}

for my region’s com.amazonaws.us-west-2.s3

cdk version
2.39.0 (build e36bfe5)

0reactions

capraquacommented, Sep 6, 2022

I am having exact same problem. Appreciate any updates. Below is my log.

8:54:22 PM | CREATE_FAILED | Custom::AWSCDK-EKS-KubernetesResource | EKSServiceA/EKS-Cl...t/Resource/Default Received response status [FAILED] from custom resource. Message returned: Error: connect ETIMEDOUT 3.0.187.238:443 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1159:16) (RequestId: d468ded6-8099-41d7-a3e3-927f71850ad3) 8:54:23 PM | UPDATE_ROLLBACK_IN_P | AWS::CloudFormation::Stack | MicroserviceStack The following resource(s) failed to create: [EKSServiceAEKSClusterAwsAuthmanifest955E58E6].

NVM, i fixed by change eks cluster to Private_with_nat, it seems the cluster needs to access to the internet.