Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[aws-eks] cannot define eks cluster and network in different stacks

See original GitHub issue

in v1.57.0 stack reference doesn’t work, everything is fine in v1.56.0

Reproduction Steps

network_sg_stack = NetworkSgStack(
    scope=app,
    id='sg-stack-' + environment,
    env=env,
    environment=environment,
    vpc_id=app.node.try_get_context(target_env)['vpc_id'],
)

eks_stack = EksStack(
    scope=app,
    id='eks-stack-' + environment,
    env=env,
    environment=environment,
    vpc=network_sg_stack.vpc,
    subnet_a_eks=network_sg_stack.subnet_c_eks,
    subnet_b_eks=network_sg_stack.subnet_b_eks,
    subnet_c_eks=network_sg_stack.subnet_c_eks,
    cluster_version=aws_eks.KubernetesVersion.V1_16
    tags=tags,
    eks_control_plane_role=iam_stack.eks_control_plane_role,
    eks_control_plane_sg=network_sg_stack.eks_control_plane_sg,
)

–>

What did you expect to happen?

What actually happened?

jsii.errors.JavaScriptError: 
  Error: 'sg-stack-develop-cdk2' depends on 'edp-eks-stack-develop-cdk2' (sg-stack-develop-cdk2 -> eks-stack-develop-cdk2/cluster/KubectlProviderSecurityGroup/Resource.GroupId). Adding this dependency (stack-develop-cdk2 -> sg-stack-develop-cdk2/eks-control-plane-sg-develop-cdk2/Resource.GroupId) would create a cyclic reference.
      at Stack._addAssemblyDependency (/tmp/jsii-kernel-mpylii/node_modules/@aws-cdk/core/lib/stack.js:415:19)
      at Object.addDependency (/tmp/jsii-kernel-mpylii/node_modules/@aws-cdk/core/lib/deps.js:47:24)
      at Stack.addDependency (/tmp/jsii-kernel-mpylii/node_modules/@aws-cdk/core/lib/stack.js:194:16)
      at resolveValue (/tmp/jsii-kernel-mpylii/node_modules/@aws-cdk/core/lib/private/refs.js:85:14)
      at Object.resolveReferences (/tmp/jsii-kernel-mpylii/node_modules/@aws-cdk/core/lib/private/refs.js:27:30)
      at Object.prepareApp (/tmp/jsii-kernel-mpylii/node_modules/@aws-cdk/core/lib/private/prepare-app.js:34:16)
      at Object.synthesize (/tmp/jsii-kernel-mpylii/node_modules/@aws-cdk/core/lib/private/synthesis.js:16:19)
      at App.synth (/tmp/jsii-kernel-mpylii/node_modules/@aws-cdk/core/lib/stage.js:78:41)
      at /home/alexey/.local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7746:51
      at Kernel._wrapSandboxCode (/home/alexey/.local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:8398:19)
      at /home/alexey/.local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7746:25
      at Kernel._ensureSync (/home/alexey/.local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:8371:20)
      at Kernel.invoke (/home/alexey/.local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7745:26)
      at KernelHost.processRequest (/home/alexey/.local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7446:28)
      at KernelHost.run (/home/alexey/.local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7384:14)
      at Immediate._onImmediate (/home/alexey/.local/lib/python3.7/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7387:37)
      at processImmediate (internal/timers.js:456:21)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "app.py", line 105, in <module>
    app.synth()
  File "/home/alexey/.local/lib/python3.7/site-packages/aws_cdk/core/__init__.py", line 11426, in synth
    return jsii.invoke(self, "synth", [options])
  File "/home/alexey/.local/lib/python3.7/site-packages/jsii/_kernel/__init__.py", line 113, in wrapped
    return _recursize_dereference(kernel, fn(kernel, *args, **kwargs))
  File "/home/alexey/.local/lib/python3.7/site-packages/jsii/_kernel/__init__.py", line 296, in invoke
    return _callback_till_result(self, response, InvokeResponse)
  File "/home/alexey/.local/lib/python3.7/site-packages/jsii/_kernel/__init__.py", line 181, in _callback_till_result
    response = kernel.sync_complete(response.cbid, None, result, response_type)
  File "/home/alexey/.local/lib/python3.7/site-packages/jsii/_kernel/__init__.py", line 339, in sync_complete
    response_type=response_type
  File "/home/alexey/.local/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 369, in sync_complete
    resp = self._process.send(_CompleteRequest(complete=request), response_type)
  File "/home/alexey/.local/lib/python3.7/site-packages/jsii/_kernel/providers/process.py", line 318, in send
    raise JSIIError(resp.error) from JavaScriptError(resp.stack)
jsii.errors.JSIIError: 'sg-stack-develop-cdk2' depends on 'eks-stack-develop-cdk2' (sg-stack-develop-cdk2 -> eks-stack-develop-cdk2/cluster/KubectlProviderSecurityGroup/Resource.GroupId). Adding this dependency (eks-stack-develop-cdk2 -> sg-stack-develop-cdk2/eks-control-plane-sg-develop-cdk2/Resource.GroupId) would create a cyclic reference.
Subprocess exited with error 1

Environment

  • **CLI Version 1.57.0
  • **Framework Version:1.57.0
  • **Node.js Version:v12.18.1
  • **OS :ubuntu 18.04
  • **Language (Version):Python 3.7.5

Other

This is 🐛 Bug Report

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:6 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
oleksii-boiko-uacommented, Aug 15, 2020

app.py

#!/usr/bin/env python3

from aws_cdk import (
    core,
    aws_eks,
    aws_ec2,
)

from eks_auth_mapping_stack import EksAuth
from eks_control_plane_stack import EksStack
from eks_worker_nodes_stack import EksWorkerNodesStack
from iam_stack import IamStack
from lib.permission_boundary import PermissionBoundaryAspect
from network_sg_stack import NetworkSgStack

app = core.App()

# Define target environment from input  cdk synth -c env=develop
target_env = app.node.try_get_context('environment')
# Define environment  variables
account = app.node.try_get_context(target_env)['account']
region = app.node.try_get_context(target_env)['region']
env = core.Environment(account=account, region=region)
environment = app.node.try_get_context(target_env)['name']
permissions_boundary = app.node.try_get_context(target_env)['permissions_boundary']
tags = {
    'Project': 'test',
    'Environment': environment,
}

# Declare stacks using CDK

network_sg_stack = NetworkSgStack(
    scope=app,
    id='test-sg-stack-' + environment,
    env=env,
    environment=environment,
    # Import values from cf output
    # core.Fn.import_value("vpc-VpcId") track this issue https://github.com/aws/aws-cdk/issues/4096
    vpc_id=app.node.try_get_context(target_env)['vpc_id'],
)

iam_stack = IamStack(
    scope=app,
    id='test-iam-stack-' + environment,
    env=env,
    environment=environment,
)

eks_stack = EksStack(
    scope=app,
    id='test-eks-stack-' + environment,
    env=env,
    environment=environment,
    vpc=network_sg_stack.vpc,
    subnet_a_eks=network_sg_stack.subnet_c_eks,
    subnet_b_eks=network_sg_stack.subnet_b_eks,
    subnet_c_eks=network_sg_stack.subnet_c_eks,
    cluster_version=aws_eks.KubernetesVersion.V1_16,  # hardcoded du to new cdk update
    tags=tags,
    eks_control_plane_role=iam_stack.eks_control_plane_role,
    eks_control_plane_sg=network_sg_stack.eks_control_plane_sg,
)

# Apply  permissions_boundary globally to all iam roles using hacky library until implemented by aws cdk
app.node.apply_aspect(PermissionBoundaryAspect(permissions_boundary))

# Synthesize!
app.synth()

network_sg_stack.py

from aws_cdk import (
    core,
    aws_ec2,
    aws_ssm,
)


class NetworkSgStack(core.Stack):
    def __init__(
            self,
            scope: core.Construct,
            id: str,
            environment: str,
            vpc_id: str,
            **kwargs,
    ) -> None:
        super().__init__(scope, id, **kwargs)

        # Fetch vpc
        self.vpc = aws_ec2.Vpc.from_lookup(
            self,
            'Vpc',
            vpc_id=vpc_id,
        )

        # Fetch Subnets

        self.subnet_a_eks = aws_ec2.Subnet.from_subnet_attributes(self, 'subnetaeks',
                                                                  availability_zone=self.region + 'a',
                                                                  subnet_id=aws_ssm.StringParameter.value_from_lookup(
                                                                      self, "/config/network/subnetaeks"))
        self.subnet_b_eks = aws_ec2.Subnet.from_subnet_attributes(self, 'subnetbeks',
                                                                  availability_zone=self.region + 'b',
                                                                  subnet_id=aws_ssm.StringParameter.value_from_lookup(
                                                                      self, "/config/network/subnetbeks"))
        self.subnet_c_eks = aws_ec2.Subnet.from_subnet_attributes(self, 'subnetceks',
                                                                  availability_zone=self.region + 'c',
                                                                  subnet_id=aws_ssm.StringParameter.value_from_lookup(
                                                                      self, "/config/network/subnetceks"))
        self.subnet_a_inter_cx = aws_ec2.Subnet.from_subnet_attributes(self, 'subnetaintercx',
                                                                       availability_zone=self.region + 'a',
                                                                       subnet_id=core.Fn.import_value(
                                                                           'vpc-SubnetIdAInterCX'))
        self.subnet_b_inter_cx = aws_ec2.Subnet.from_subnet_attributes(self, 'subnetbintercx',
                                                                       availability_zone=self.region + 'b',
                                                                       subnet_id=core.Fn.import_value(
                                                                           'vpc-SubnetIdBInterCX'))
        self.subnet_c_inter_cx = aws_ec2.Subnet.from_subnet_attributes(self, 'subnetcintercx',
                                                                       availability_zone=self.region + 'c',
                                                                       subnet_id=core.Fn.import_value(
                                                                           'vpc-SubnetIdCInterCX'))

        # Control plane
        self.eks_control_plane_sg = aws_ec2.SecurityGroup(
            scope=self,
            id='control-plane',
            security_group_name=id + '-control-plane',
            vpc=self.vpc,
            allow_all_outbound=True,
            description='EKS control plane SG for cluster'
        )

        # SG
        self.eks_worker_sg = aws_ec2.SecurityGroup(
            scope=self,
            id='worker',
            security_group_name=id + '-worker',
            vpc=self.vpc,
            allow_all_outbound=True,
            description='EKS node SG for eks cluster in %s' % environment,
        )

        # Allow all within the worker nodes
        self.eks_worker_sg.add_ingress_rule(
            peer=self.eks_worker_sg,
            connection=aws_ec2.Port.all_traffic(),
        )

        # Allow ports 0-65535 from control plane to workers
        self.eks_worker_sg.add_ingress_rule(
            peer=self.eks_control_plane_sg,
            connection=aws_ec2.Port.tcp_range(start_port=0, end_port=65535),
        )
        self.eks_control_plane_sg.add_egress_rule(
            peer=self.eks_worker_sg,
            connection=aws_ec2.Port.tcp_range(start_port=0, end_port=65535),
        )

        # Allow port 443 from workers to control plane
        self.eks_control_plane_sg.add_ingress_rule(
            peer=self.eks_worker_sg,
            connection=aws_ec2.Port.tcp(port=443),
        )

eks_control_plane_stack.py

from aws_cdk import (
    core,
    aws_ec2,
    aws_eks,
    aws_ssm,
    aws_iam,
)
from aws_cdk.aws_eks import KubernetesVersion


class EksStack(core.Stack):
    def __init__(
            self,
            scope: core.Construct,
            id: str,
            environment: str,
            vpc: aws_ec2.IVpc,
            subnet_a_eks: aws_ec2.ISubnet,
            subnet_b_eks: aws_ec2.ISubnet,
            subnet_c_eks: aws_ec2.ISubnet,
            cluster_version: KubernetesVersion,
            eks_control_plane_role: aws_iam.IRole,
            eks_control_plane_sg: aws_ec2.ISecurityGroup,
            **kwargs,
    ) -> None:
        super().__init__(scope, id, **kwargs)

        # Control plane  had to put legacy for now
        self.cluster = aws_eks.Cluster(
            scope=self,
            id='cluster',
            cluster_name="cluster-" + environment,
            kubectl_enabled=True,
            # endpoint_access=aws_eks.EndpointAccess.PUBLIC_AND_PRIVATE,
            default_capacity=0,
            vpc=vpc,
            vpc_subnets=[aws_ec2.SubnetSelection(subnets=[subnet_a_eks, subnet_b_eks, subnet_c_eks])],
            # issue with 3 subnet
            version=cluster_version,
            security_group=eks_control_plane_sg,
            role=eks_control_plane_role,
        )
        self.cluster_cert_param = aws_ssm.StringParameter(
            scope=self,
            id='clusterCertParam' + environment,
            string_value=self.cluster.cluster_certificate_authority_data,
            description='Some user-friendly description',
            parameter_name='/' + environment + '/config/' 'clustercertparam',
        )

iam_stack.py

import typing

from aws_cdk import (
    core,
    aws_iam,
)

_eks_worker_role_base_policies = (
    aws_iam.ManagedPolicy.from_aws_managed_policy_name('AmazonEKSWorkerNodePolicy'),
    aws_iam.ManagedPolicy.from_aws_managed_policy_name('AmazonEKS_CNI_Policy'),
    aws_iam.ManagedPolicy.from_aws_managed_policy_name('AmazonEC2ContainerRegistryReadOnly'),
)


class IamStack(core.Stack):
    def __init__(
            self,
            scope: core.Construct,
            id: str,
            environment: str,
            **kwargs,
    ) -> None:
        super().__init__(scope, id, **kwargs)

        # Creating Amazon EKS Cluster Role
        managed_policy = aws_iam.ManagedPolicy.from_aws_managed_policy_name('AmazonEKSClusterPolicy')
        cluster_name = 'cluster-' + environment

        self.eks_control_plane_role = aws_iam.Role(
            scope=self,
            id='control-plane',
            role_name=id + '-control-plane',
            assumed_by=aws_iam.ServicePrincipal('eks.amazonaws.com'),
            managed_policies=[managed_policy],
        )

        self.eks_worker_role = aws_iam.Role(
            scope=self,
            id='worker-role',
            role_name=id + '-worker-role',
            path='/eks/',
            assumed_by=aws_iam.ServicePrincipal('ec2.amazonaws.com'),
            managed_policies=list(_eks_worker_role_base_policies),
        )
        self.eks_worker_role.add_to_policy(
            aws_iam.PolicyStatement(
                resources=['*'],
                actions=['secretsmanager:GetSecretValue']
            )
        )

        core.Tag.add(
            scope=self.eks_worker_role,
            key='eks/%s/type' % cluster_name,
            value='node'
        )

        # Creating Amazon EKS External Secret Role
        self.external_secrets_role = aws_iam.Role(
            scope=self,
            id='external-secrets',
            role_name=id + '-external-secrets',
            assumed_by=aws_iam.ServicePrincipal('ec2.amazonaws.com'),
        )

        self.external_secrets_role.add_to_policy(
            aws_iam.PolicyStatement(
                resources=['*'],
                actions=['secretsmanager:GetSecretValue', 'secretsmanager:ListSecrets',
                         'secretsmanager:GetResourcePolicy', 'secretsmanager:DescribeSecret',
                         'secretsmanager:ListSecretVersionIds', 'ssm:GetParameters',
                         'ssm:GetParameter', 'ssm:GetParametersByPath',
                         'ssm:GetParameterHistory']
            )
        )

        # Creating Roles which will be mapped to k8s rback group with different rights
        principal = aws_iam.AccountRootPrincipal()

        self.eks_admin_team_role = eks_roles(
            scope=self,
            id='eks-admin-team',
            role_name=id + '-admin',
            k8s_username='admin',
            k8s_groups=['system:masters'],
            cluster_name=cluster_name,
            principal=principal,
        )

        self.eks_dev_team_role = eks_roles(
            scope=self,
            id='eks-dev-team',
            role_name=id + '-dev-team',
            k8s_username='dev-team',
            k8s_groups=['dev-team'],
            cluster_name=cluster_name,
            principal=principal,
        )

        self.eks_dev_team_advanced_role = eks_roles(
            scope=self,
            id='eks-dev-team-advanced',
            role_name=id + '-dev-team-advanced',
            k8s_username='dev-team-advanced',
            k8s_groups=['dev-team-advanced'],
            cluster_name=cluster_name,
            principal=principal,
        )

        self.cluster_users = [self.eks_admin_team_role, self.eks_dev_team_role, self.eks_dev_team_advanced_role]


def eks_roles(
        scope: core.Construct,
        id: str,
        role_name: str,
        k8s_username: str,
        k8s_groups: typing.List[str],
        cluster_name: str,
        principal: aws_iam.IPrincipal,
) -> aws_iam.Role:
    cluster_access = aws_iam.PolicyDocument(
        statements=[
            aws_iam.PolicyStatement(
                actions=['eks:DescribeCluster'],
                resources=['*'],
            )
        ]
    )

    role = aws_iam.Role(
        scope=scope,
        id=id,
        path='/eks/',
        role_name=role_name,
        assumed_by=principal,
        inline_policies={
            'cluster-access': cluster_access
        }
    )

    core.Tag.add(
        scope=role,
        key='eks/%s/type' % cluster_name,
        value='user'
    )
    core.Tag.add(
        scope=role,
        key='eks/%s/username' % cluster_name,
        value=k8s_username,
    )
    core.Tag.add(
        scope=role,
        key='eks/%s/groups' % cluster_name,
        value=','.join(k8s_groups),
    )
0reactions
iliapolocommented, Aug 15, 2020

@alexey-boyko Not archive. Just paste the relevant resources. Specifically, i’m interested in exactly how are you instantiating the subnets and the VPC in the NetworkSgStack.

Read more comments on GitHub >

github_iconTop Results From Across the Web

How do I resolve cluster creation errors in Amazon EKS?
Resolution · Recreate the cluster in a different Availability Zone · Confirm that you have the correct IAM permissions to create a cluster....
Read more >
Amazon EKS networking - AWS Documentation
Your Amazon EKS cluster is created in a VPC. Pod networking is provided by the Amazon VPC Container Network Interface (CNI) plugin.
Read more >
Troubleshoot upgrade fails with my Amazon EKS cluster
1. Open the Amazon EKS console in the Region where you created your cluster. · 2. Select Clusters from the sidebar. · 3....
Read more >
Troubleshoot eksctl issues with EKS cluster and node group
If you created the Amazon EKS cluster with PrivateOnly networking, then AWS CloudFormation can't create public subnets. This means that export ...
Read more >
Create Amazon EKS clusters and node groups without a route ...
1. In the following config file, update the AWS Region and three PrivateOnly subnets that you created in the Create a VPC for...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

[core] Unable to Synthesize CDK in 1.57: Assets must be defined indirectly within a "Stage" or an "App" scope

Next page

[aws-lambda-python] Unify `entry` field betwen aws-lambda-nodejs and aws-lambda-python