[aws-lambda-nodejs] a single lambda creation takes 30s every time

tl;dr;

Node.js lambda are slow to build because CDK uses docker and mounting a complete project inside docker is super slow on macos.

There’s work being done on Docker side, but it will probably always be very slow for now. There’s work being done on CDK to allow using a local parcel instead of inside docker, see https://github.com/aws/aws-cdk/pull/9632.

For an alternative solution, I created https://github.com/vvo/aws-lambda-nodejs-webpack which uses webpack locally and is a lot faster

---

Hi there, when using aws-lambda-nodejs, every cdk synth will take up to 50s, even for a very simple setup. I understand aws-lambda-nodejs uses docker + parcel but even that, with the right cache, performance should not be an issue on successive runs because of Docker / parcel cache. So I guess something is weird here.

As discussed over email with you @jogold, I am creating a new issue that you can reproduce on your side.

Reproduction Steps

I created a repository: https://github.com/vvo/aws-lambda-nodejs-performance

The stack code is:

const cdk = require("@aws-cdk/core");
const lambda = require("@aws-cdk/aws-lambda-nodejs");

class HelloCdkStack extends cdk.Stack {
  constructor(scope, id, props) {
    super(scope, id, props);
    new lambda.NodejsFunction(this, "my-handler");
  }
}

module.exports = { HelloCdkStack };

Reproduction:

git clone https://github.com/vvo/aws-lambda-nodejs-performance.git
cd aws-lambda-nodejs-performance
npm install
time cdk synth

# [... cdk output]
cdk synth  1.82s user 0.63s system 8% cpu 27.730 total

Here it’s 27 seconds, but I have seen it go up to 50s.

Error Log

No error message

Environment

• CLI Version : 1.51.0
• Framework Version: 1.51.0
• Node.js Version: 12.18.2
• **OS : MacOS 10.15.5
• Language (Version): JavaScript

Other

I’d like to debug further and understand why, maybe, the cache is not used fully but not sure how to do that.

---

This is 🐛 Bug Report

[hoegertn]

For the other repo the first run was about 3 minutes and subsequent runs about 40 - 50 seconds.

The first-run delay seems to be the sync that Docker does for the folder. It copies the whole folder to the hyperkit vm.

The asset-output error did not happen this time. It seems related to what @vvo described earlier.

[vvo]

when mounting the volumes? when Parcel runs? (+when content is written to /asset-output?)

As you know, I did a lot of testing on this, results:

On docker stable version: both actions are extremely slow. Mounting is slow (node_modules, lots of files) and parcel is slow (lots of I/O). On my current project, cdk synth can take up to two minutes, at every run. 30s was for an almost empty Node.js project (no dependencies).

On docker edge/beta version: only the mounting is slow (still 5x faster than stable), afterwards parcel is as fast as native/without docker. On docker edge, :delegated currently uses mutagen by default (may or may not be the end of the story once it reaches docker stable, it might be a different flag)

:delegated is always the fastest for any operation on docker mac os, as seen here:

• https://www.amazee.io/blog/post/docker-on-mac-performance-docker-machine-vs-docker-for-mac
• https://www.jeffgeerling.com/blog/2020/revisiting-docker-macs-performance-nfs-volumes

Docker edge makes the mounting “faster”. But as for my testing goes, 20s to mount a real world Node.js project will never be acceptable… By real world I mean my project, approx 600MB node_modules, which is not even big compared to more “enterprise” projects I have seen.

We can wait for docker to be faster (but it has been “slow” on macOS filesystems mounting for 3 years already) OR we can make it so it does not use docker on mac. I understand this is not an area you want to go but unfortunately, the road you took will, for now, lead to bad performance on mac for anyone trying to build and deploy Node.js lambdas on AWS using this tool.

This will impact A LOT of people. Data I found is from the 2018 Node.js developer survey: MacOS is leading the Node.js developer environment with 41% of users (Windows at 24%), https://nodejs.org/en/user-survey-report/#Primary-OS-Distro. This means that for most people, the experience is not good for now. And those are the people that will try cdk and abandon it if they have the choice.

For comparison, I tried https://github.com/pulumi/pulumi and it was instant almost for every command, I still want to use CDK because I do not want/need multi-cloud. And pulumi has some weird defaults as for file organization and how it builds lambdas.

Other areas to test:

• yarn v2 performance. What might be impacting performance may not be the SIZE of the project but rather the number of files of the project. And unfortunately, node_modules are a LOOOOOT of files to mount. While new versions of yarn have fewer files in node_modules if I remember well (“pnp”?)
• as for running parcel outside of docker, npx is GREAT! https://github.com/npm/npx#readme. It just means that on first launch it would install it, without even installing it globally and then executes it. npx is bundled with Node.js since 8.2.0

What about using cached

We tried, I tried, results are worse.

Other tests done:

• read only + delegated on BUNDLING_INPUT_DIR: no change versus just delegated
• read only + cached on BUNDLING_INPUT_DIR: no change versus just delegated
• cached on BUNDLING_INPUT_DIR: no change versus just delegated
• again, when using delegated or cached, the slow part comes from mounting, parcel is fast as seen in the docker output: parcel takes always 5s while the whole docker run takes 20s