[aws-eks] attach cluster security group to self managed nodes to allow free communication between all node groups

Self managed node groups created with the cluster.addAutoScalingGroupCapacity function does not get a shared security group allowing all traffic between different node groups. Also traffic from managed node groups to self managed node groups are only allowed for tcp port 1025-65535. This causes DNS traffic on port 53 to be blocked between node groups which becomes an issue if you have core-dns pods running on a subset of your self managed node groups as groups without core-dns pods can’t do DNS lookups.

The above issue does not exist with eksctl as it creates a shared security group that is used for all self managed and managed node groups, which allow all traffic between them.

Reproduction Steps

Create an EKS cluster with the aws-eks module’s cluster construct. Add two self managed node groups with the addAutoScalingGroupCapacity function. Add a managed node group with the addNodegroupCapacity function.

Configure core-dns to run on only one of the self managed node groups.

Start a pod on the other self managed node group that tries to do a nslookup on any domain e.g. kubernetes, or github.com. It fails as it can’t connect to the cluster IP for the kube-dns service.

Repeat the above step for a managed node group.

Note: DNS lookup works across managed node groups, if there is a core-dns pod running in any of them, as they all use the same security group, which allows all traffic internally.

What did you expect to happen?

DNS lookups should be possible anywhere in the cluster, regardless of where the core-dns pods are running.

What actually happened?

DNS lookups was not possible due to blocked traffic by the node security groups.

Environment

• CLI Version : aws-cli/2.0.44
• Framework Version: cdk 1.66.0 (build 459488d)
• Node.js Version: v12.18.4
• OS : Darwin/19.6.0
• Language (Version): Version 3.9.7

Other

Working scenarios for DNS lookups (pod doing the lookup on the left, core-dns pod on the right): Self managed node group 1 -> Self managed node group 1 Managed node group 1 -> Managed node group 1 Managed node group 1 -> Managed node group 2

Failing scenarios for DNS lookups: Self managed node group 1 -> Self managed node group 2 Managed node group 1 -> Self managed node group 1

---

This is 🐛 Bug Report

[dirknilius]

Not sure…Do you maybe have a minimal reproduction test case in code that I can run?

@iliapolo I could extract the essentials from my stack and create a demo that you can deploy. I will post the link when I’m done.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.