(pipelines): 'npx cdk' is failing in version 1.103.0, causing CDK pipelines synth action to fail

CDK Pipeline, when using NPM simple synth action, runs:

• npm ci to install dependencies (including dev dependencies) according to package-lock.json
• npx cdk synth to synthesize the app

If aws-cdk is not listed in package’s dependencies, the first command will not install it. Then the npx command will not find aws-cdk locally and installs it. However, it will install the latest version available, so release is not properly versioned. Also, from version 1.103.0 it fails with following message: Cannot find module '.../aws-sdk/scripts/check-node-version.js'.

I would like to raise the following questions:

• Is it correct and desirable that the pipeline installs dev dependencies? Wouldn’t it be cleaner to only install prod ones using npm ci --production? In that case, aws-cdk would need to become production dependency of each project
• Would it be better to run npx --no-install cdk synth as synth command, to ensure the pipeline requires the aws-cdk dependecy and cannot just silently use the latest cdk cli?

Anyways, I would assume that for convenience, npx cdk ... should still work without the package being previously installed.

Reproduction Steps

For consistency, I am using aws/codebuild/standard:5.0 docker image that CDK pipelines use for synthesis, can be find here: https://github.com/aws/aws-codebuild-docker-images/tree/master/ubuntu/standard/5.0 and build using docker build -t aws/codebuild/standard:5.0 .

What did you expect to happen?

Previous version (1.102.0) works fine:

tomas@arkham:~$ docker run --rm -it --entrypoint='' aws/codebuild/standard:5.0 npx --package aws-cdk@1.102.0 cdk version
npx: installed 188 in 9.121s
1.102.0 (build a75d52f)
***************************************************
*** Newer version of CDK is available [1.103.0] ***
*** Upgrade recommended                         ***
***************************************************

What actually happened?

Latest version (1.103.0) fails:

tomas@arkham:~$ docker run --rm -it --entrypoint='' aws/codebuild/standard:5.0 npx cdk version
internal/modules/cjs/loader.js:883
  throw err;
  ^

Error: Cannot find module '/root/.npm/_npx/1/lib/node_modules/cdk/node_modules/aws-cdk/node_modules/aws-sdk/scripts/check-node-version.js'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
    at Function.Module._load (internal/modules/cjs/loader.js:725:27)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
    at internal/main/run_main_module.js:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! aws-sdk@2.903.0 postinstall: `node scripts/check-node-version.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the aws-sdk@2.903.0 postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2021-05-12T10_51_22_468Z-debug.log
Install for [ 'cdk@latest' ] failed with code 1

Environment

• CDK CLI Version : 1.103.0
• Framework Version: N/A
• Node.js Version: v14.15.4
• OS : CodeBuild standard:5.0 container image
• Language (Version): TypeScript

---

This is 🐛 Bug Report

[rix0rrr]

env NPM_CONFIG_UNSAFE_PERM=true npx my-demo-postinstall-plugin@1.0.0

[rix0rrr]

Ha-haaah here is the magic:

$ npm set unsafe-perm true