(s3-deployment): Support large s3 bucket deployments

We want to enable users to deploy large amounts of data onto s3 buckets. Currently, such deployments frequently fail because we are limited by the lambda execution environment:

• Deployment size should be no more than 500MB because that is the amount of disk space available on the /tmp mount of the lambda.
• sync duration has to be less than 15 minutes because this is a hard execution timeout for lambda.

See https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

Use Case

Users use the s3-deployment module to deploy static websites to an s3 bucket. On many occasions, the contents of the website directory can be quite big, and contain many files.

See:

• https://github.com/aws/aws-cdk/issues/7571
• https://github.com/aws/aws-cdk/issues/7316
• https://github.com/aws/aws-cdk/issues/6893

Proposed Solution

No clear solution to this problem yet, this issue is created so we discuss various approaches.

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

---

This is a 🚀 Feature Request

[aiden-sobey]

@ababakanian the memory limit needs to be increased on the custom resource that deploys to the S3 bucket behind your CloudFront distribution.

import * as deployment from 'aws-cdk-lib/aws-s3-deployment';

        new deployment.BucketDeployment(this, 'deploy', {
            destinationBucket: this.bucket,
            distribution: this.distribution,
            distributionPaths: ['/*'], // Invalidate the CloudFront cache on upload
            sources: [
                deployment.Source.asset(buildPath),
            ],
            memoryLimit: 1024,
        });

You want the last line in that, memoryLimit.

[iliapolo]

Some work has been done to solve the 500MB limitation by allowing to attach an EFS volume. See https://github.com/aws/aws-cdk/pull/12361.

That PR has gone stale and is now closed, but if anyone want to pick it up i’ll be happy to merge it.