(lambda): EFS and Lambda create cyclic reference when deployed in separate stacks

What is the problem?

EFS (particularly AccessPoint) and Lambda create cyclic reference when deployed in separate stacks.

Very related: #5760, #10942, #11245. All of which are P1. Check the reproducible code in those issues as well.

Reproduction Steps

Here is a working example:

export class TheEfsLambdaStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const vpc = new ec2.Vpc(this, 'Vpc', {
      maxAzs: 2, // Default is all AZs in the region
    });

    // Create a file system in EFS to store information
    const fs = new efs.FileSystem(this, 'FileSystem', {
      vpc,
      removalPolicy: cdk.RemovalPolicy.DESTROY
    });

    const accessPoint = fs.addAccessPoint('AccessPoint',{
      createAcl: {
        ownerGid: '1001',
        ownerUid: '1001',
        permissions: '750'
      },
      path:'/export/lambda',
      posixUser: {
        gid: '1001',
        uid: '1001'
      }
    });

    // This lambda function is given access to our EFS File System
    const efsLambda = new lambda.Function(this, 'efsLambdaFunction', {
      runtime: lambda.Runtime.PYTHON_3_8,
      code: lambda.Code.fromAsset('lambda-fns'), 
      handler: 'message_wall.lambda_handler',
      vpc: vpc,
      filesystem: lambda.FileSystem.fromEfsAccessPoint(accessPoint, '/mnt/msg')
    });

    // defines an API Gateway Http API resource backed by our "efsLambda" function.
    let api = new apigw.HttpApi(this, 'Endpoint', {
      defaultIntegration: new integrations.LambdaProxyIntegration({
        handler: efsLambda
      })
    });

   new cdk.CfnOutput(this, 'HTTP API Url', {
     value: api.url ?? 'Something went wrong with the deploy'
   });
  }
}

The code breaks if Function is deployed in separate stack and the AccessPoint is passed in props, with the following error:

/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/stack.ts:395
      throw new Error(`'${target.node.path}' depends on '${this.node.path}' (${cycle.join(', ')}). Adding this dependency (${reason}) would create a cyclic reference.`);
            ^
Error: 'Stack2' depends on 'Stack' ("Stack2/lambda/lambda/ServiceRole/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget1", "Stack2/lambda/lambda/ServiceRole/DefaultPolicy/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget1", "Stack2/lambda/lambda/SecurityGroup/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget1", "Stack2/lambda/lambda/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget1", "Stack2/lambda/lambda/ServiceRole/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget2", "Stack2/lambda/lambda/ServiceRole/DefaultPolicy/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget2", "Stack2/lambda/lambda/SecurityGroup/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget2", "Stack2/lambda/lambda/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget2", "Stack2/lambda/lambda/ServiceRole/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget3", "Stack2/lambda/lambda/ServiceRole/DefaultPolicy/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget3", "Stack2/lambda/lambda/SecurityGroup/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget3", "Stack2/lambda/lambda/Resource" depends on "Stack/efs/FileSystem/EfsMountTarget3", "Stack2/lambda/lambda/Resource" depends on "Stack/efs/FileSystem/EfsSecurityGroup/from StackefsFileSystemEfsSecurityGroup2F9415D6:ALL PORTS", "Stack2/lambda/lambda/Resource" depends on "Stack/efs/FileSystem/EfsSecurityGroup/from StackfunctionSecurityGroupD8C55079:2049", "Stack2/lambda/lambda/Resource" depends on "Stack/efs/FileSystem/EfsSecurityGroup/from Stack2clipmodeldownloadSecurityGroupB2C2BEB1:2049"). Adding this dependency (Stack -> Stack2/lambda/lambda/SecurityGroup/Resource.GroupId) would create a cyclic reference.
    at Stack._addAssemblyDependency (/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/stack.ts:395:13)
    at Object.addDependency (/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/deps.ts:52:20)
    at Stack.addDependency (/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/stack.ts:266:5)
    at resolveValue (/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/private/refs.ts:100:12)
    at Object.resolveReferences (/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/private/refs.ts:30:24)
    at Object.prepareApp (/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/private/prepare-app.ts:30:3)
    at Object.synthesize (/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/private/synthesis.ts:32:3)
    at App.synth (/Users/julian/work/project/node_modules/aws-cdk-lib/core/lib/stage.ts:90:23)
    at Object.<anonymous> (/Users/julian/work/project/bin/infra.ts:134:5)
    at Module._compile (internal/modules/cjs/loader.js:1072:14)

For more code insight check similar issues: #5760, #10942, #11245.

I can add simple self-contained reproducible code but it should be simple enough to infer.

What did you expect to happen?

Should work. We can pass Vpc in props across stacks, why EFS and others fail? The error is confusing as well.

What actually happened?

Fails as described above.

CDK CLI Version

2.10

Framework Version

No response

Node.js Version

14 LTS

OS

Mac

Language

Typescript

Language Version

TS 4.x

Other information

No response

[corymhall]

It looks like the issue is with this code. We already have a dependency Lambda -> AccessPoint and this is trying to create the security group rules in the AccessPoint stack (AccessPoint -> Lambda)

https://github.com/aws/aws-cdk/blob/bb8d6f6bf5941b76ef0590c99fe8e26440e09c18/packages/%40aws-cdk/aws-lambda/lib/function.ts#L1007-L1011

This should be updated to be something like this so that the security group rules are added in the lambda stack.

if (props.filesystem) {
  if (props.filesystem.config.connections) {
    this.connections.allowTo(props.filesystem.config.connections, props.filesystem.config.connections.defaultPort);
  }
}

[moltar]

Thanks @fivepapertigers for this escape hatch!

Here’s the same in TS:

    fileSystem.connections.securityGroups.forEach((fssg) => {
      fssg.node.findAll().forEach((child)=> {
        if (child instanceof CfnSecurityGroupIngress &&
          lambda.connections.securityGroups.some(({ securityGroupId }) => securityGroupId === child.sourceSecurityGroupId)) {
          fssg.node.tryRemoveChild(child.node.id);
        }
      });
    });

    lambda.connections.allowToDefaultPort(fileSystem);