sam sync runs into infinite error loop `cannot find esbuild` when npm dependencies are changed and does no longer create node_modules even on clean re-run

Description:

When using sam sync there is no dependencies layer being created. However, when I run sam deploy first and then sam sync there is an initial node_modules folder in a layer. But changing dependencies afterwards will remove the layer, not create a new one and fail with an error as esbuild can no longer be found.

Steps to reproduce:

follow https://aws.amazon.com/blogs/compute/building-typescript-projects-with-aws-sam-cli/

Observed result:

🟢 sam init

sam-init.log

file structure

└──     events/
 │  └────     event.json
 └──     hello-world/
 │  └────     tests/
 │  │  └────     unit/
 │  │  │  └────     test-handler.test.ts
 │  ├────     .eslintignore
 │  ├────     .eslintrc.js
 │  ├────     .npmignore
 │  ├────     .prettierrc.js
 │  ├────     app.ts
 │  ├────     jest.config.ts
 │  ├────     package.json
 │  └────     tsconfig.json
 ├──     .gitignore
 ├──     README.md
 └──     template.yaml

---

🟢 sam build

sam-build.log

I did not need to change template.yaml as it already contained the Metadata part mentioned in the docs

file structure

└──     .aws-sam/
 │  └────     build/
 │  │  └────     HelloWorldFunction/
 │  │  │  ├────     app.js
 │  │  │  └────     app.js.map
 │  │  └────     template.yaml
 │  └────     build.toml
 └──     events/
 │  └────     event.json
 └──     hello-world/
 │  └────     tests/
 │  │  └────     unit/
 │  │  │  └────     test-handler.test.ts
 │  ├────     .eslintignore
 │  ├────     .eslintrc.js
 │  ├────     .npmignore
 │  ├────     .prettierrc.js
 │  ├────     app.ts
 │  ├────     jest.config.ts
 │  ├────     package.json
 │  └────     tsconfig.json
 ├──     .gitignore
 ├──     README.md
 └──     template.yaml

---

🟢 sam deploy --guided

sam-deploy.log

no changes in file structure

---

❌ sam sync --stack-name node-modules-test-6 --watch (fails after dependencies are changed)

sam-sync.log

• calling the lambda now works as expected and node_modules exist for expected layer id.
• changing code also works fine and results in changed return message from lambda function.

But installing a dependency (@aws-sdk/client-s3) will result in an error loop due to missing esbuild. The /deps/xyz folder will be empty.

file structure

└──     .aws-sam/
 │  └────     auto-dependency-layer/
 │  │  └────     HelloWorldFunction/
 │  │  │  ├────     app.js
 │  │  │  └────     app.js.map
 │  │  └────     HelloWorldFunction19d43fc4DepLayer/
 │  │  │  └────     nodejs/
 │  │  │  └────     AWS_SAM_CLI_README
 │  │  ├────     nested_template.yaml
 │  │  └────     template.yaml
 │  └────     build/
 │  │  └────     HelloWorldFunction/
 │  │  │  ├────     app.js
 │  │  │  └────     app.js.map
 │  │  └────     template.yaml
 │  └────     cache/
 │  └────     deps/
 │  │  └────     6f9fcb11-f1f8-439c-a68e-7a57d7bf4835/
 │  │  │  └────     node_modules/ << ✅ contains expected modules
 │  └────     build.toml
 └──     events/
 │  └────     event.json
 └──     hello-world/
 │  └────     node_modules/
 │  └────     tests/
 │  │  └────     unit/
 │  │  │  └────     test-handler.test.ts
 │  ├────     .eslintignore
 │  ├────     .eslintrc.js
 │  ├────     .npmignore
 │  ├────     .prettierrc.js
 │  ├────     app.ts
 │  ├────     jest.config.ts
 │  ├────     package.json
 │  └────     tsconfig.json
 ├──     .gitignore
 ├──     README.md
 ├──     samconfig.toml
 └──     template.yaml

Log excerpt

Code sync encountered an error.
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflows/nodejs_npm_esbuild/actions.py", line 230, in execute
    version = self.subprocess_esbuild.run(args, cwd=self.scratch_dir)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflows/nodejs_npm_esbuild/esbuild.py", line 93, in run
    invoke_esbuild = [self.esbuild_binary()] + args
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflows/nodejs_npm_esbuild/esbuild.py", line 64, in esbuild_binary
    raise EsbuildExecutionError(message="cannot find esbuild")
aws_lambda_builders.workflows.nodejs_npm_esbuild.esbuild.EsbuildExecutionError: Esbuild Failed: cannot find esbuild

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflow.py", line 302, in run
    action.execute()
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflows/nodejs_npm_esbuild/actions.py", line 232, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: Esbuild Failed: cannot find esbuild

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/samcli/lib/build/app_builder.py", line 740, in _build_function_in_process
    builder.build(
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/aws_lambda_builders/builder.py", line 164, in build
    return workflow.run()
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflow.py", line 96, in wrapper
    func(self, *args, **kwargs)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflow.py", line 309, in run
    raise WorkflowFailedError(workflow_name=self.NAME, action_name=action.NAME, reason=str(ex))
aws_lambda_builders.exceptions.WorkflowFailedError: NodejsNpmEsbuildBuilder:EsbuildCheckVersion - Esbuild Failed: cannot find esbuild

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/samcli/lib/sync/sync_flow_executor.py", line 335, in _sync_flow_execute_wrapper
    dependent_sync_flows = sync_flow.execute()
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/samcli/lib/sync/sync_flow.py", line 291, in execute
    self.gather_resources()
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/samcli/lib/sync/flows/zip_function_sync_flow.py", line 93, in gather_resources
    build_result = builder.build()
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/samcli/lib/build/app_builder.py", line 219, in build
    return ApplicationBuildResult(build_graph, build_strategy.build())
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/samcli/lib/build/build_strategy.py", line 510, in build
    result.update(super().build())
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/samcli/lib/build/build_strategy.py", line 75, in build
    result.update(self._build_functions(self._build_graph))
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/samcli/lib/build/build_strategy.py", line 85, in _build_functions
    function_build_results.update(self.build_single_function_definition(build_definition))
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/samcli/lib/build/build_strategy.py", line 520, in build_single_function_definition
    return self._incremental_build_strategy.build_single_function_definition(build_definition)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/samcli/lib/build/build_strategy.py", line 419, in build_single_function_definition
    return self._delegate_build_strategy.build_single_function_definition(build_definition)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/samcli/lib/build/build_strategy.py", line 156, in build_single_function_definition
    result = self._build_function(
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/samcli/lib/build/app_builder.py", line 654, in _build_function
    return self._build_function_in_process(
  File "/opt/homebrew/Cellar/aws-sam-cli/1.40.1/libexec/lib/python3.8/site-packages/samcli/lib/build/app_builder.py", line 757, in _build_function_in_process
    raise BuildError(wrapped_from=ex.__class__.__name__, msg=str(ex)) from ex
samcli.lib.build.exceptions.BuildError: NodejsNpmEsbuildBuilder:EsbuildCheckVersion - Esbuild Failed: cannot find esbuild
Syncing Lambda Function HelloWorldFunction...
Manifest is not changed for af9e866d-e660-457a-8861-371c48d0b747, running incremental build
Building codeuri: /Users/bastian/development/awssam/node-modules-test-3/hello-world runtime: nodejs14.x metadata: {'BuildMethod': 'esbuild', 'BuildProperties': {'Minify': True, 'Target': 'es2020', 'Sourcemap': True, 'EntryPoints': ['app.ts']}} architecture: x86_64 functions: ['HelloWorldFunction']
Running NodejsNpmEsbuildBuilder:CopySource
Running NodejsNpmEsbuildBuilder:CopySource
Running NodejsNpmEsbuildBuilder:EsbuildCheckVersion

---

⚠️ delete .aws-sam and re-run sam sync --stack-name node-modules-test-6 --watch

sam-sync-retry.log

This won’t create any new layers / update node_modules and keep the initial layer in the cloud for running the function, which will throw as the dependency cannot be found.

The folder deps will remain empty.

Workaround

• using --no-dependency-layer seems to work fine

Expected result:

• dependency layers are being re-created when node_modules change
• esbuild should be available after dependency changes and not cause sync to error
• initial sam sync should probably also create a layer without having to run sam deploy first (or it should be explicitly mentioned)

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Machine: 
- Mac mini (M1, 2020)
- Chip Apple M1
- Memory 16 GB
- Startup Disk Macintosh HD

System:
macOS version: 12.3
Node manager:  fnm
Node version:  v14.18.2
SAM version:   SAM CLI, version 1.40.1
AWS-Region:    eu-central-1

Additional notes

• it would be very helpful if the following lines could be disabled from --debug output (was like 2/3 of all logs during sync)

Found the same SyncFlow in queue. Skip adding.
No Parameters detected in the template
There is no customer defined id or cdk path defined for resource HelloWorldFunction, so we will use the resource logical id as the resource id
There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the resource logical id as the resource id
Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction is HelloWorldFunction

[passionkind]

Finally had a chance to try it again and yes, it works as expected 🙏👏 thanks a lot!

[passionkind]

A little update:

I’ve just set up my brand new Mac. I didn’t copy/paste/restore as I usually do. I wanted to have a really clean start, so I reinstalled everything manually and paid special attention to not follow any bad practices just because the system is supposed to run as I wish right now (am I the only one who’s falling into that pit sometimes? 🙈)

And I can confirm that the result is the exact same: Esbuild Failed: cannot find esbuild.

What systems are you guys using at AWS to run in. Or do you meanwhile experience the same issue?