[aws-eks] eks cluster connections refers to wrong security group

After creating an EKS cluster with a default Managed Node Group via the aws-eks module’s Cluster() constructor, I noticed that if I tried to allow ingress to the nodes and pods on the Node Group via cluster.connections that it failed to work. This is because the security group associated with cluster.connections is the secondary security group that’s being assigned to the cluster via its resourcesVpcConfig, instead of the primary cluster security group that’s returned by the EKS CreateCluster API.

EKS Managed Node Groups and the pods that run on them are associated with the cluster security group that’s returned by the EKS cluster API, not any additional security groups that a customer may configure.

Reproduction Steps

const cluster = new eks.Cluster(this, clusterName, {
    clusterName,
    version: eks.KubernetesVersion.V1_17,
    mastersRole: accessRole,
    vpc,
    vpcSubnets: [
      { subnetType: ec2.SubnetType.PRIVATE },
    ],
    defaultCapacity: 2
  }
));

cluster.connections.allowFrom(externalClient.connections, ec2.Port.allTcp());

What did you expect to happen?

I expected an ingress rule to be associated with the cluster security group associated with my cluster that allowed ingress from all TCP ports from my external client’s security group.

What actually happened?

An ingress rule was added to the additional security group associated with my cluster that allowed ingress from all TCP ports from my external client’s security group. Unfortunately this has no effect on the pods running in the cluster.

Environment

• CLI Version : 1.56.0
• Framework Version: 1.56.0
• Node.js Version: 12.18.2
• OS : MacOS Catalina
• Language (Version): TypeScript (3.8.3)

Other

The following code provides an interim workaround:

  new ec2.Connections({
    securityGroups: [
      ec2.SecurityGroup.fromSecurityGroupId(this, 
           `${clusterName}ClusterSecurityGroup`, cluster.clusterSecurityGroupId)
      ]
  }).allowFrom(externalClient.connections, ec2.Port.allTcp());

---

This is 🐛 Bug Report

[otterley]

Hi @iliapolo - as we discussed elsewhere, I think it should be the other way around. Most customers will use the auto-generated security group since it applies to the cluster, any Managed Node group instances, and the pods; and so I think it makes sense to return that one in cluster.connections instead.

The documentation doesn’t make this clear, but the security group that is provided in resourcesVpcConfig at the API level is optional. The typical case for using it is when you already have a set of non-managed nodes in a security group you’ve already created, and you want to connect those nodes to the cluster. In that case, if you want to access the connections object, you’ll use the connections from the pre-existing EC2 instances. It’s not needed for managed node groups, nor is it needed if you want to put non-managed nodes in the same security group as the one generated by the EKS service when you create a cluster.

[acim]

We have this problem as well. In our use case we want to allow access to Vault from Kubernetes cluster. The only way to do it now it to use clusterSecurityGroupId to fetch the security group and then to call allowFrom method on the Vault’s connections. This works, but it creates dependency from Vault to Kubernetes stack which is bad because Vault stack should be created much earlier and really should not depend on anything else (run on it’s own EC2).

If this security group behind clusterSecurityGroupId becomes available under connections or nodesConnections, whatever, we could call allowTo directly and point to Vault. I believe this would not create the dependency because we have similar case with other stacks where connections is available (with the needed security group).

Additional (control plane) security group associated to connections at the moment is also must have, but cluster group should also be directly accessible. And I agree with @otterley that primary group should be associated to connections property and secondary one to controlPlaneConnections or whatever. This is BC, I know, but it sounds logical that primary group is under connections and additional ones under something else.