(assets): Docker based bundling forces redeployment of assets when python environment changes.

❓ General Issue

I am using a Docker-based bundling option to install necessary dependencies and create a Lambda Layer. The deployment succeeds. If I try to deploy a few more times - the bundling is initiated, however, in the end, it states “No changes”. And this is expected behavior, since I haven’t changed anything, no changes are introduced.

Now the real hocus-pocus happens is when I run the deployment command in a CI/CD pipeline which installs a fresh environment every time. The CDK gets confused and even though no changes to the asset are introduced, it redeploys my layer every single time.

The Question

What do I have to do in order to have deployments only when the changes are introduced to the asset?

Environment

• CDK CLI Version: 1.75.0
• Module Version: 1.75.0
• Node.js Version: v15.1.0
• OS: Ubuntu
• Language (Version): Python3

Edit 1: Bundling asset layer:

LayerVersion(
    scope=scope,
    id=id,
    layer_version_name=name,
    code=Code.from_asset(
        self.get_source_path(),
        asset_hash_type=AssetHashType.BUNDLE,
        bundling=BundlingOptions(
            image=BundlingDockerImage.from_registry('python:3.9'),
            command=[
                'bash', '-c', ' && '.join([
                    'pip install boto3 --upgrade -t /tmp/asset-output/python',
                    'find /tmp/asset-output -type f -name "*.py[co]" -delete',
                    'find /tmp/asset-output -type d -name "__pycache__" -delete',
                    'cp -R /tmp/asset-output/. /asset-output/.',
                    'cp -R /asset-input/. /asset-output/.',
                    'ls -la /asset-output/python/.'
                ])
            ]
        )
    ),
    compatible_runtimes=self.runtimes()
)

Edit 2: Very important update !!!

This is a sequence of steps that reproduce “redeployment” behavior.

• I just tried to create two virtual environments.
• Installed exactly the same dependencies.
• Sourced the first environment.
• Initiated deployment.
• Deployment deployed my layer.
• Initiated deployment once more.
• Deployment was not executed with a reason “no changes”.
• Now I sourced the second environment.
• Initiated the deployment and what a surprise - the deployment redeployed my layer. Even though no changes were introduced to my asset and both virtual environments were identical.

What happened? How AWS CDK decides when it is time to execute the redeployment?

[laimonassutkus]

@jogold @redbaron @pgollucci @eladb I have traced down the problem. Apparently, the python creates __pycache__ folders which result in unpredictable hashes. To avoid this, run a custom script to delete these types of folders or make sure the environment variable is set:

export PYTHONDONTWRITEBYTECODE=1

The custom script to run before executing cdk deploy:

find . -type f -name "*.py[co]" -delete
find . -type d -name "__pycache__" -exec rm -rf {} +
find . -type d -name "*.dist-info" -exec rm -rf {} +
find . -type d -name "*.egg-info" -exec rm -rf {} +

With that said - in order to be completely in control of reproducing consistent asset builds, you should always use AssetHashType.BUNDLE and provide custom build commands to build and clean the asset.

Hope this helps for future similar problems to other people.

[redbaron]

@laimonassutkus , you can try adding __pycache__ to dockerignore, if I remember correctly CDK respects it