(eks): Support isolated VPCs

Provisioning clusters inside an isolated vpc (i.e no internet access) is not currently supported. This is because the lambda functions that operate the cluster need to invoke the EKS service, which does not offer a VPC endpoint.

See https://github.com/aws/containers-roadmap/issues/298

Use Case

We’ve seen users mentioning their environment uses an isolated VPC.

Other

Adding some information here to possibly facilitate alternative approaches.

If you have a proxy setup, you can inject proxy information to the handlers via custom environment variables.

const proxy = "https://proxy.mycompany.com:8080/”;
new eks.Cluster(this, 'Cluster', {
  ...,

  kubectlEnvironment: {
    HTTPS_PROXY: proxy,
  },

  clusterHandlerEnvironment: {
    HTTPS_PROXY: proxy
  }
})

Also, following is a list of AWS services that our Lambda handlers interact with in order to operate the cluster. All of these services offer a VPC endpoint except for EKS.

• Lambda
• Step Functions
• CloudFormation
• STS
• S3
• EKS

Related: https://github.com/aws/aws-cdk/issues/10036

Once EKS does offer a VPC endpoint, it would be nice if we just provision the necessary endpoints given if we identify that the VPC does not have internet access (internet gateway, NAT).

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

---

This is a 🚀 Feature Request

[BowlesCR]

In my scenario, my “isolated” subnets aren’t really isolated from the internet as I use a TGW to route traffic via an egress network. If you try for private and natGateways=0, CDK insists you call them isolated. If you call them isolated, you can’t put EKS on them.

Is there a workaround to this, or could there be some sort of “I know what I’m doing” override added?

[BowlesCR]

Sure thing Stacktrace (file paths lightly sanitized):

jsii.errors.JavaScriptError: 
  Error: There are no 'Private' subnet groups in this VPC. Available types: Isolated,Public
      at Vpc.selectSubnetObjectsByType (/tmp/jsii-kernel-Jsmxfu/node_modules/@aws-cdk/aws-ec2/lib/vpc.js:206:19)
      at Vpc.selectSubnetObjects (/tmp/jsii-kernel-Jsmxfu/node_modules/@aws-cdk/aws-ec2/lib/vpc.js:172:28)
      at Vpc.selectSubnets (/tmp/jsii-kernel-Jsmxfu/node_modules/@aws-cdk/aws-ec2/lib/vpc.js:59:30)
      at /tmp/jsii-kernel-Jsmxfu/node_modules/@aws-cdk/aws-eks/lib/cluster.js:265:77
      at Array.map (<anonymous>)
      at new Cluster (/tmp/jsii-kernel-Jsmxfu/node_modules/@aws-cdk/aws-eks/lib/cluster.js:265:59)
      at /tmp/tmpyat3mqfa/lib/program.js:2720:58
      at Kernel._wrapSandboxCode (/tmp/tmpyat3mqfa/lib/program.js:3148:24)
      at Kernel._create (/tmp/tmpyat3mqfa/lib/program.js:2720:34)
      at Kernel.create (/tmp/tmpyat3mqfa/lib/program.js:2461:29)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./CDK-Infrastructure/app.py", line 382, in <module>
    main()
  File "./CDK-Infrastructure/app.py", line 338, in main
    eks_stack = EksStack(
  File "./CDK-Infrastructure/.env/lib64/python3.9/site-packages/jsii/_runtime.py", line 83, in __call__
    inst = super().__call__(*args, **kwargs)
  File "./CDK-Infrastructure/cdk_infrastructure/cdk_eks/cdk_eks_stack.py", line 48, in __init__
    self.cluster = eks.Cluster(
  File "./CDK-Infrastructure/.env/lib64/python3.9/site-packages/jsii/_runtime.py", line 83, in __call__
    inst = super().__call__(*args, **kwargs)
  File "./CDK-Infrastructure/.env/lib64/python3.9/site-packages/aws_cdk/aws_eks/__init__.py", line 7895, in __init__
    jsii.create(Cluster, self, [scope, id, props])
  File "./CDK-Infrastructure/.env/lib64/python3.9/site-packages/jsii/_kernel/__init__.py", line 265, in create
    response = self.provider.create(
  File "./CDK-Infrastructure/.env/lib64/python3.9/site-packages/jsii/_kernel/providers/process.py", line 348, in create
    return self._process.send(request, CreateResponse)
  File "./CDK-Infrastructure/.env/lib64/python3.9/site-packages/jsii/_kernel/providers/process.py", line 330, in send
    raise JSIIError(resp.error) from JavaScriptError(resp.stack)
jsii.errors.JSIIError: There are no 'Private' subnet groups in this VPC. Available types: Isolated,Public
Subprocess exited with error 1

cdk_eks_stack.py Line 48:

        self.cluster = eks.Cluster(
            self,
            "Cluster",
            cluster_name=cluster_name,
            vpc=vpc,
            version=eks.KubernetesVersion.V1_18,
            default_capacity=0,
            endpoint_access=eks.EndpointAccess.PRIVATE,
            masters_role=adminRole,
            secrets_encryption_key=secrets_key,
            security_group=security_group,
            # vpc_subnets=vpc.isolated_subnets,
        )