(aws-lambda-nodejs): redeploy triggers when no code changed although root directory changed

If we have a function with node_modules and an included sub module, cdk deploy triggers a re-deploy without any code changes to a lambda function, but with a file path change to the root source directory.

Assume a lambda function that requires a module and that has a submodule. If the root directory of the project changes (for example, is in a second cloned git repository or is on another machine like a GitHub workflow triggered via on push), even when there have been no code changes, cdk diff shows there exist changes and cdk deploy pushes changed code up to cloud.

Reproduction Steps

The public git repository listed below contains a small hello world CDK project I was using to learn CDK. In the process, I noticed re-deploys in GitHub workflows (on ubuntu) even though I had already deployed from my local machine (macOS) prior to a push origin. After quite a bit of sleuthing, I narrowed it down to a file path difference. See steps below for reproducing (works on my mac). Note: a change of root directory triggers the issue.

> git clone https://github.com/codeedog/aws-test-hello-world.git
> cd aws-test-hello-world
> npm ci
> (cd layers/hworld/nodejs && npm ci)
# Above line is to make sure the layer builds, properly, but it's not a part of this bug
> cdk bootstrap
> cdk deploy
> cd ..
> mv aws-test-hello-world aws-test && cd $_
> cdk diff
# There will be differences
> cdk deploy
# It will deploy again, even though only the name of the root directory changed

What did you expect to happen?

The particular location of a root source tree should not affect the analysis and determination of changes and a need for a new compilation and code push. I expected the change analysis tool to determine there were no changes and respond with (no changes).

What actually happened?

cdk diff noted differences in the function that contained node modules when there were no changes. cdk deploy pushed a new version to the cloud.

Environment

• CDK CLI Version : 1.107.0 (build 52c4434)
• Framework Version:
• Node.js Version: v14.16.1
• OS : macOS Catalina 10.15.7 (19H524) & Ubuntu (20.04.2 LTS, on GitHub)
• Language (Version): Typescript (3.9.9 from package-lock.json)

Other

Code Repo Details

Example code lives in github here. See Example 2 on line 28. The self-contained lambda function uses axios to grab a webpage. axios uses a submodule called follow-redirects. Unsure if the submodule is required or any module will do.

I was playing with an independent lambda function (relies on nothing), a lambda function that includes some modules and a lambda function that uses one layer. The middle one was giving me aggravation, and that was how I tracked down the problem. The third function was also creating a problem until I explicitly declared the sub module in the layer’s package.json file.

The Layer creation code in aws-lambda correctly determines how to package the layer and is not confused by root directory changes. Meanwhile, aws-lambda is confused by root directory changes.

I did a diff -r on a pair of deployments with only the root directory changed. Here’s a result:

diff -r cdk.out/asset.098edaf0d9b823e56cdbf9ebd0a22d3d032686eecde90f45cb790b4677cb6b65/ cdk.out/asset.b150aa06cfd9fd726b029e33c74a1b055cc1dd757e7a1aef173fc2d9f9ff69a9/

diff -r cdk.out/asset.098.../node_modules/axios/package.json cdk.out/asset.b15.../node_modules/axios/package.json
5c5
<       "/Users/_path_/aws-test/cdk.out/bundling-temp-f710..."
---
>       "/Users/_path_/aws/cdk.out/bundling-temp-2b44..."
29c29
<   "_where": "/Users/_path_/aws-test/cdk.out/bundling-temp-f710...",
---
>   "_where": "/Users/_path_/aws/cdk.out/bundling-temp-2b44...",

diff -r cdk.out/asset.098.../node_modules/follow-redirects/package.json cdk.out/asset.b15.../node_modules/follow-redirects/package.json
5c5
<       "/Users/_path_/aws-test/cdk.out/bundling-temp-f710..."
---
>       "/Users/_path_/aws/cdk.out/bundling-temp-2b44..."
30c30
<   "_where": "/Users/_path_/aws-test/cdk.out/bundling-temp-f710...",
---
>   "_where": "/Users/_path_/aws/cdk.out/bundling-temp-2b44...",

---

This is 🐛 Bug Report

[jogold]

@nija-at looks like this is https://github.com/npm/npm/issues/10393 and https://github.com/npm/npm/issues/12110.

For some reason, NPM stores absolute paths in some underscored keys when modules are installed…

We use a temporary bundling directory for bundling and node modules installation. Its name includes a cache key that is influenced by the absolute path of the asset being bundled. Maybe we can make this path relative to the asset out dir?

Note that the issue only arises when using bundling.nodeModules with a NodejsFunction.

[nija-at]

On the other hand, running the following command yields,

⇒  diff -r aws-test-hello-world/cdk.out/asset.1e4b33621164c1f9a4c433a589a70fd739ea3eb37bc35cf26ecdf6c83e6cfa88 another-clone/cdk.out/asset.031f95bb18f4e0bc06683401f3b62311f0619644566d284611a491e44a211fb9
diff -r aws-test-hello-world/cdk.out/asset.1e4b33621164c1f9a4c433a589a70fd739ea3eb37bc35cf26ecdf6c83e6cfa88/node_modules/.package-lock.json another-clone/cdk.out/asset.031f95bb18f4e0bc06683401f3b62311f0619644566d284611a491e44a211fb9/node_modules/.package-lock.json
2c2
<   "name": "bundling-temp-f4f9391c05f8a28805387c4b769408ba7af7d22d0c4fc47bca74675ce2628595",
---
>   "name": "bundling-temp-c7e3dbdb1a29b07424b8bac2c249b004a32aa50bff88dc65d8c15cf6d7811301",
diff -r aws-test-hello-world/cdk.out/asset.1e4b33621164c1f9a4c433a589a70fd739ea3eb37bc35cf26ecdf6c83e6cfa88/package-lock.json another-clone/cdk.out/asset.031f95bb18f4e0bc06683401f3b62311f0619644566d284611a491e44a211fb9/package-lock.json
2c2
<   "name": "bundling-temp-f4f9391c05f8a28805387c4b769408ba7af7d22d0c4fc47bca74675ce2628595",
---
>   "name": "bundling-temp-c7e3dbdb1a29b07424b8bac2c249b004a32aa50bff88dc65d8c15cf6d7811301",

This difference is coming from -

https://github.com/aws/aws-cdk/blob/163e8122db994d0bea7077f025876dbeac490ead/packages/%40aws-cdk/core/lib/asset-staging.ts#L411

@eladb, @jogold - is it expected that the same repo cloned into different locations should yield different asset hashes? Feels incorrect.