Intermittant TLS handshake failures in sam build with container build

Description:

Seeing intermittent failures when using the --use-container option of sam build. No changes in configuration caused this, it just started happening. SAM is being installed via https://github.com/aws-actions/setup-sam and I also attempted to pin the version to the latest one.

This appears to either be an issue with the way aws-sam-cli is doing a TLS handshake with ECR, or it is a failure with ECR.

Steps to reproduce:

1. Use https://github.com/aws-actions/setup-sam actions
2. Use the use-container option.

Observed result:

Docker image is sometimes not found due to 500 error / TLS error.

raise DockerImagePullFailedException(str(ex)) from ex
samcli.local.docker.manager.DockerImagePullFailedException: 500 Server Error: Internal Server Error ("Get "https://public.ecr.aws/v2/": net/http: TLS handshake timeout")

Full output, including build commands:

> npm run build && sam build --parallel --cached --use-container


> <OMITTED>@0.0.1 build
> npm run clean && tsc


> <OMITTED>@0.0.1 clean
> rm -fr build

Starting Build use cache
Starting Build inside a container
Cache is invalid, running build and copying resources to function build definition of <OMITTED>

Fetching public.ecr.aws/sam/build-nodejs14.x:latest-x86_64 Docker container image.............................................................................................Cache is invalid, running build and copying resources to layer build definition of c23e0d67-4bb0-4ae6-ad5e-caffb11bda90
Building layer 'RuntimeDependenciesLayer'
For container layer build, first compatible runtime is chosen as build target for container.
.Cache is invalid, running build and copying resources to layer build definition of <OMITTED>
Building layer '<OMITTED>'
.For container layer build, first compatible runtime is chosen as build target for container.
.................................................................................................................................................................................................................................................................................................................................................
Mounting /home/runner/work/<OMITTED>/build as /tmp/samcli/source:ro,delegated inside runtime container
package.json file not found. Continuing the build without dependencies.
Running NodejsNpmBuilder:CopySource

Fetching public.ecr.aws/sam/build-nodejs14.x:latest-x86_64 Docker container image......
Mounting /home/runner/work/<OMITTED> as /tmp/samcli/source:ro,delegated inside runtime container
Running CustomMakeBuilder:CopySource
Running CustomMakeBuilder:MakeBuild
Current Artifacts Directory : /tmp/samcli/artifacts
Traceback (most recent call last):
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.[10](https://github.com/<OMITTED>/runs/5880392319?check_suite_focus=true#step:7:10)/site-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.35/images/create?tag=latest-x86_64&fromImage=public.ecr.aws%2Fsam%2Fbuild-nodejs14.x%3Alatest-x86_64

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/local/docker/manager.py", line 160, in pull_image
    result_itr = self.docker_client.api.pull(image_name, tag=tag, stream=True, decode=True)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/docker/api/image.py", line 415, in pull
    self._raise_for_status(response)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("Get "https://public.ecr.aws/v2/": net/http: TLS handshake timeout")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/local/docker/manager.py", line 85, in create
    self.pull_image(image_name)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/local/docker/manager.py", line 163, in pull_image
    raise DockerImagePullFailedException(str(ex)) from ex
samcli.local.docker.manager.DockerImagePullFailedException: 500 Server Error: Internal Server Error ("Get "https://public.ecr.aws/v2/": net/http: TLS handshake timeout")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/_temp/setup-sam-6sS74t/bin/sam", line 8, in <module>
    sys.exit(cli())
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/telemetry/metric.py", line 166, in wrapped
    raise exception  # pylint: disable=raising-bad-type
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/telemetry/metric.py", line 124, in wrapped
    return_value = func(*args, **kwargs)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/utils/version_checker.py", line 41, in wrapped
    actual_result = func(*args, **kwargs)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/cli/main.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/commands/build/command.py", line 174, in cli
    do_cli(
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/commands/build/command.py", line 251, in do_cli
    ctx.run()
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/commands/build/build_context.py", line 191, in run
    build_result = builder.build()
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/build/app_builder.py", line 219, in build
    return ApplicationBuildResult(build_graph, build_strategy.build())
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/build/build_strategy.py", line 364, in build
    async_results = self._async_context.run_async()
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/utils/async_utils.py", line 131, in run_async
    return run_given_tasks_async(self._async_tasks, event_loop)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/utils/async_utils.py", line 90, in run_given_tasks_async
    return event_loop.run_until_complete(_run_given_tasks_async(tasks, event_loop, executor))
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/utils/async_utils.py", line 58, in _run_given_tasks_async
    raise result
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/build/build_strategy.py", line 543, in build_single_layer_definition
    return self._cached_build_strategy.build_single_layer_definition(layer_definition)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/build/build_strategy.py", line 304, in build_single_layer_definition
    build_result = self._delegate_build_strategy.build_single_layer_definition(layer_definition)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/build/build_strategy.py", line 205, in build_single_layer_definition
    layer.full_path: self._build_layer(
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/build/app_builder.py", line 524, in _build_layer
    self._build_function_on_container(
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/build/app_builder.py", line 816, in _build_function_on_container
    self._container_manager.run(container)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/local/docker/manager.py", line [11](https://github.com/<OMITTED>/runs/5880392319?check_suite_focus=true#step:7:11)5, in run
    self.create(container)
  File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/local/docker/manager.py", line 88, in create
    raise DockerImagePullFailedException(
samcli.local.docker.manager.DockerImagePullFailedException: Could not find public.ecr.aws/sam/build-nodejs[14](https://github.com/<OMITTED>/runs/5880392319?check_suite_focus=true#step:7:14).x:latest-x86_[64](https://github.com/<OMITTED>/runs/5880392319?check_suite_focus=true#step:7:64) image locally and failed to pull it from docker.
Error: Process completed with exit code 1.

Expected result:

The build should consistently not fail due to a Docker error.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Linux (Github Actions)
2. sam --version: any
3. AWS region: us-east-1 or us-east-2

[thatarchguy]

I am seeing these failures on 2/3 or more of the sam builds when deploying to us-east-1 or us-east-2.

I am seeing this exact behavior. 2/3 or more sam builds in us-east-2 are failing.

[danielrhodes]

@qingchm I’m wondering if there’s any follow up here. Is there a better place I could file a ticket or any additional context I can provide?