(aws-lambda-nodejs): Use esbuild API instead of CLI for bundling

What is the problem?

If a dependency has a .node dependency, for example ssh2 in node_modules/ssh2/lib/protocol/crypto.js has

binding = require('./crypto/build/Release/sshcrypto.node');,

this will have esbuild fail the bundling with

error: No loader is configured for ".node" files: ../node_modules/ssh2/lib/protocol/crypto/build/Release/sshcrypto.node.

In https://github.com/evanw/esbuild/issues/1051 they provide a plugin, but sadly plugins can’t be used with binary esbuild.

Reproduction Steps

Run npx cdk init app --language=typescript to create sample app

Add new NodejsFunction(this, 'testfunc', { entry: 'lib/testfunc/index.ts' }); to test stack

Add to file lib/testfunc/index.ts:

import { Client } from "ssh2";

const handler = async () => {
  const conn = new Client();

  conn.end();
};

export { handler };

Add to file lib/testfunc/package.json:

{
  "dependencies": {
    "ssh2": "^1.5.0"
  }
}

Run npm run cdk synth

What did you expect to happen?

It bundling the example code

What actually happened?

$ npm run cdk synth

> cdktest@0.1.0 cdk
> cdk "synth"

Bundling asset CdktestStack/testfunc/Code/Stage...
✘ [ERROR] No loader is configured for ".node" files: lib/testfunc/node_modules/cpu-features/build/Release/cpufeatures.node

    lib/testfunc/node_modules/cpu-features/lib/index.js:1:24:
      1 │ const binding = require('../build/Release/cpufeatures.node');
        ╵                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

✘ [ERROR] No loader is configured for ".node" files: lib/testfunc/node_modules/ssh2/lib/protocol/crypto/build/Release/sshcrypto.node

    lib/testfunc/node_modules/ssh2/lib/protocol/crypto.js:30:20:
      30 │   binding = require('./crypto/build/Release/sshcrypto.node');
         ╵                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

2 errors

/home/azatoth/tmp/cdktest/node_modules/aws-cdk-lib/core/lib/asset-staging.ts:395
      throw new Error(`Failed to bundle asset ${this.node.path}, bundle output is located at ${bundleErrorDir}: ${err}`);
            ^
Error: Failed to bundle asset CdktestStack/testfunc/Code/Stage, bundle output is located at /home/azatoth/tmp/cdktest/cdk.out/bundling-temp-43706a6374f4884c6246dca414315bc291a0eff7496c6eda3da7968ee92f4c47-error: Error: bash -c npx --no-install esbuild --bundle "/home/azatoth/tmp/cdktest/lib/testfunc/index.ts" --target=node14 --platform=node --outfile="/home/azatoth/tmp/cdktest/cdk.out/bundling-temp-43706a6374f4884c6246dca414315bc291a0eff7496c6eda3da7968ee92f4c47/index.js" --external:aws-sdk run in directory /home/azatoth/tmp/cdktest exited with status 1
    at AssetStaging.bundle (/home/azatoth/tmp/cdktest/node_modules/aws-cdk-lib/core/lib/asset-staging.ts:395:13)
    at AssetStaging.stageByBundling (/home/azatoth/tmp/cdktest/node_modules/aws-cdk-lib/core/lib/asset-staging.ts:243:10)
    at stageThisAsset (/home/azatoth/tmp/cdktest/node_modules/aws-cdk-lib/core/lib/asset-staging.ts:134:35)
    at Cache.obtain (/home/azatoth/tmp/cdktest/node_modules/aws-cdk-lib/core/lib/private/cache.ts:24:13)
    at new AssetStaging (/home/azatoth/tmp/cdktest/node_modules/aws-cdk-lib/core/lib/asset-staging.ts:159:44)
    at new Asset (/home/azatoth/tmp/cdktest/node_modules/aws-cdk-lib/aws-s3-assets/lib/asset.ts:72:21)
    at AssetCode.bind (/home/azatoth/tmp/cdktest/node_modules/aws-cdk-lib/aws-lambda/lib/code.ts:180:20)
    at new Function (/home/azatoth/tmp/cdktest/node_modules/aws-cdk-lib/aws-lambda/lib/function.ts:350:29)
    at new NodejsFunction (/home/azatoth/tmp/cdktest/node_modules/aws-cdk-lib/aws-lambda-nodejs/lib/function.ts:50:5)
    at new CdktestStack (/home/azatoth/tmp/cdktest/lib/cdktest-stack.ts:9:5)
Subprocess exited with error 1

CDK CLI Version

2.8.0 (build 8a5eb49)

Framework Version

No response

Node.js Version

v14.18.3

OS

Linux

Language

Typescript

Language Version

No response

Other information

No response

[shellscape]

In many cases, using the esbuild API instead of the CLI will open up the plugins configuration for ESBuild to the users.

[wprl]

Brainstorming, it seems like mostly what is needed is an option to override this command to use a custom build script: https://github.com/aws/aws-cdk/blob/v2.50.0/packages/%40aws-cdk/aws-lambda-nodejs/lib/bundling.ts#L180 and also to copy the custom build script to the Docker image when building via Docker…

For reference, the issue as seen from the esbuild side: https://github.com/evanw/esbuild/issues/884