sam local invoke adds empty AWS_SESSION_TOKEN environment variable

Update:

The problem is caused by the presence of an empty AWS_SESSION_TOKEN in the environment of the process being run in the docker container. This may or may not result in an X-Amz-Security-Token being added to calls to AWS services, depending on the implementation details of the SDK and version being used. In turn, an empty X-Amz-Security-Token may or may not get flagged as an invalid token depending on the HTTP protocol used (and possibly the service being accessed)

Description:

Some APIs don’t work with the credential provided by the sam local invoke (and, presumably, sam local start-api). The APIs seem to differ by language. This is possibly the root cause behind bugs such as #3071 and #2963. The error reported (when it is reported) is The security token included in the request is invalid

I’ve attached a sample project with two simple SAM apps. One is in Node/JavaScript, the other is in Ruby. They both contain two simple functions – one that lists all S3 buckets, and one that lists all OpsWorks stacks. In both, the S3 buckets can be listed. But only the JavaScript one retrieves the stacks – the Ruby version can’t create the client correctly.

Steps to reproduce:

1. Unpack the provided sam_local_bug.zip file.
2. Go into sam_local_bug\node_version
3. Run sam build && sam local invoke S3Test. This should pass.
4. Run sam build && sam local invoke OpsworksTest. This should pass.
5. Change to the sam_local_bug\ruby_version
6. Run sam build && sam local invoke S3Test. This should pass.
7. Run sam build && sam local invoke OpsworksTest. This should fail.

Observed result:

The output should look like this:

2021-07-26 23:09:32,970 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-07-26 23:09:32,971 | Using config file: samconfig.toml, config environment: default
2021-07-26 23:09:32,971 | Expand command line arguments to:
2021-07-26 23:09:32,971 | --template_file=<mydir>/sam_local_bug/ruby_version/.aws-sam/build/template.yaml --function_logical_id=OpsworksTest --no_event --layer_cache_basedir=<mydir>/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1 
2021-07-26 23:09:32,971 | local invoke command is called
2021-07-26 23:09:32,975 | No Parameters detected in the template
2021-07-26 23:09:32,994 | 3 stacks found in the template
2021-07-26 23:09:32,994 | No Parameters detected in the template
2021-07-26 23:09:33,010 | 3 resources found in the stack 
2021-07-26 23:09:33,010 | No Parameters detected in the template
2021-07-26 23:09:33,026 | Found Serverless function with name='S3Test' and CodeUri='S3Test'
2021-07-26 23:09:33,026 | --base-dir is not presented, adjusting uri S3Test relative to <mydir>/sam_local_bug/ruby_version/.aws-sam/build/template.yaml
2021-07-26 23:09:33,026 | Found Serverless function with name='OpsworksTest' and CodeUri='OpsworksTest'
2021-07-26 23:09:33,026 | --base-dir is not presented, adjusting uri OpsworksTest relative to <mydir>/sam_local_bug/ruby_version/.aws-sam/build/template.yaml
2021-07-26 23:09:33,033 | Found one Lambda function with name 'OpsworksTest'
2021-07-26 23:09:33,033 | Invoking opsworksTest.lambda_handler (ruby2.7)
2021-07-26 23:09:33,033 | No environment variables found for function 'OpsworksTest'
2021-07-26 23:09:33,033 | Environment variables overrides data is standard format
2021-07-26 23:09:33,033 | Loading AWS credentials from session with profile 'None'
2021-07-26 23:09:33,042 | Resolving code path. Cwd=<mydir>/sam_local_bug/ruby_version/.aws-sam/build, CodeUri=<mydir>/sam_local_bug/ruby_version/.aws-sam/build/OpsworksTest
2021-07-26 23:09:33,042 | Resolved absolute path to code is <mydir>/sam_local_bug/ruby_version/.aws-sam/build/OpsworksTest
2021-07-26 23:09:33,042 | Code <mydir>/sam_local_bug/ruby_version/.aws-sam/build/OpsworksTest is not a zip/jar file
2021-07-26 23:09:33,054 | Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-ruby2.7:rapid-1.26.0.

2021-07-26 23:09:33,054 | Mounting <mydir>/sam_local_bug/ruby_version/.aws-sam/build/OpsworksTest as /var/task:ro,delegated inside runtime container
2021-07-26 23:09:33,314 | Starting a timer for 3 seconds for function 'OpsworksTest'
START RequestId: 5707bb42-bbcf-4292-81b4-1769abc51162 Version: $LATEST
Error raised from handler method
{
  "errorMessage": "The security token included in the request is invalid.",
  "errorType": "Function<Aws::OpsWorks::Errors::UnrecognizedClientException>",
  "stackTrace": [
    "/var/runtime/gems/aws-sdk-core-3.114.3/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'",
    "/var/runtime/gems/aws-sdk-core-3.114.3/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'",
    "/var/runtime/gems/aws-sdk-core-3.114.3/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'",
    "/var/runtime/gems/aws-sdk-core-3.114.3/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'",
    "/var/runtime/gems/aws-sdk-core-3.114.3/lib/seahorse/client/plugins/request_callback.rb:71:in `call'",
    "/var/runtime/gems/aws-sdk-core-3.114.3/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'",
    "/var/runtime/gems/aws-sdk-core-3.114.3/lib/seahorse/client/plugins/response_target.rb:24:in `call'",
    "/var/runtime/gems/aws-sdk-core-3.114.3/lib/seahorse/client/request.rb:72:in `send_request'",
    "/var/runtime/gems/aws-sdk-opsworks-1.32.0/lib/aws-sdk-opsworks/client.rb:3534:in `describe_stacks'",
    "/var/task/opsworksTest.rb:33:in `lambda_handler'"
  ]
}
END RequestId: 5707bb42-bbcf-4292-81b4-1769abc51162
REPORT RequestId: 5707bb42-bbcf-4292-81b4-1769abc51162	Init Duration: 0.10 ms	Duration: 1576.66 ms	Billed Duration: 1600 ms	Memory Size: 128 MB	Max Memory Used: 128 MB	
2021-07-26 23:09:35,062 | Cleaning all decompressed code dirs
2021-07-26 23:09:35,062 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '5fc6b3af-07d4-4594-9e80-f074b0d03661', 'installationId': '0673bfa5-1b1f-4f86-bfb2-9b8350b6c438', 'sessionId': 'b9137f15-9a01-4e18-9900-6b3f303c0457', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.11', 'samcliVersion': '1.26.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'duration': 2091, 'exitReason': 'success', 'exitCode': 0}}]}
2021-07-26 23:09:35,888 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
{"errorMessage":"The security token included in the request is invalid.","errorType":"Function<Aws::OpsWorks::Errors::UnrecognizedClientException>","stackTrace":["/var/runtime/gems/aws-sdk-core-3.114.3/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'","/var/runtime/gems/aws-sdk-core-3.114.3/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'","/var/runtime/gems/aws-sdk-core-3.114.3/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'","/var/runtime/gems/aws-sdk-core-3.114.3/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'","/var/runtime/gems/aws-sdk-core-3.114.3/lib/seahorse/client/plugins/request_callback.rb:71:in `call'","/var/runtime/gems/aws-sdk-core-3.114.3/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'","/var/runtime/gems/aws-sdk-core-3.114.3/lib/seahorse/client/plugins/response_target.rb:24:in `call'","/var/runtime/gems/aws-sdk-core-3.114.3/lib/seahorse/client/request.rb:72:in `send_request'","/var/runtime/gems/aws-sdk-opsworks-1.32.0/lib/aws-sdk-opsworks/client.rb:3534:in `describe_stacks'","/var/task/opsworksTest.rb:33:in `lambda_handler'"]}2021-07-26 23:09:35,889 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-07-26 23:09:35,889 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId': '29321452-4b76-4d24-b1d1-b1d53be778ef', 'installationId': '0673bfa5-1b1f-4f86-bfb2-9b8350b6c438', 'sessionId': 'b9137f15-9a01-4e18-9900-6b3f303c0457', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.11', 'samcliVersion': '1.26.0', 'runtimes': ['ruby2.7']}}]}
2021-07-26 23:09:36,690 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

Expected result:

Something more like this (for the JavaScript version):

2021-07-26 23:06:53,426 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-07-26 23:06:53,427 | Using config file: samconfig.toml, config environment: default
2021-07-26 23:06:53,427 | Expand command line arguments to:
2021-07-26 23:06:53,427 | --template_file=<mydir>/sam_local_bug/node_version/.aws-sam/build/template.yaml --function_logical_id=OpsworksTest --no_event --layer_cache_basedir=<mydir>/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1 
2021-07-26 23:06:53,427 | local invoke command is called
2021-07-26 23:06:53,440 | No Parameters detected in the template
2021-07-26 23:06:53,459 | 3 stacks found in the template
2021-07-26 23:06:53,459 | No Parameters detected in the template
2021-07-26 23:06:53,476 | 3 resources found in the stack 
2021-07-26 23:06:53,476 | No Parameters detected in the template
2021-07-26 23:06:53,492 | Found Serverless function with name='S3Test' and CodeUri='S3Test'
2021-07-26 23:06:53,492 | --base-dir is not presented, adjusting uri S3Test relative to <mydir>/sam_local_bug/node_version/.aws-sam/build/template.yaml
2021-07-26 23:06:53,492 | Found Serverless function with name='OpsworksTest' and CodeUri='OpsworksTest'
2021-07-26 23:06:53,492 | --base-dir is not presented, adjusting uri OpsworksTest relative to <mydir>/sam_local_bug/node_version/.aws-sam/build/template.yaml
2021-07-26 23:06:53,499 | Found one Lambda function with name 'OpsworksTest'
2021-07-26 23:06:53,499 | Invoking src/handlers/opsworksTest.opsworksTest (nodejs14.x)
2021-07-26 23:06:53,499 | No environment variables found for function 'OpsworksTest'
2021-07-26 23:06:53,500 | Environment variables overrides data is standard format
2021-07-26 23:06:53,500 | Loading AWS credentials from session with profile 'None'
2021-07-26 23:06:53,508 | Resolving code path. Cwd=<mydir>/sam_local_bug/node_version/.aws-sam/build, CodeUri=<mydir>/sam_local_bug/node_version/.aws-sam/build/OpsworksTest
2021-07-26 23:06:53,508 | Resolved absolute path to code is <mydir>/sam_local_bug/node_version/.aws-sam/build/OpsworksTest
2021-07-26 23:06:53,508 | Code <mydir>/sam_local_bug/node_version/.aws-sam/build/OpsworksTest is not a zip/jar file
2021-07-26 23:06:53,521 | Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-nodejs14.x:rapid-1.26.0.

2021-07-26 23:06:53,521 | Mounting <mydir>/sam_local_bug/node_version/.aws-sam/build/OpsworksTest as /var/task:ro,delegated inside runtime container
2021-07-26 23:06:53,806 | Starting a timer for 60 seconds for function 'OpsworksTest'
START RequestId: 989cd964-4474-4255-bcda-c0265a9b4045 Version: $LATEST
END RequestId: 989cd964-4474-4255-bcda-c0265a9b4045
REPORT RequestId: 989cd964-4474-4255-bcda-c0265a9b4045	Init Duration: 0.14 ms	Duration: 2310.45 ms	Billed Duration: 2400 ms	Memory Size: 128 MB	Max Memory Used: 128 MB	
2021-07-26 23:06:56,283 | Cleaning all decompressed code dirs
2021-07-26 23:06:56,284 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '86949428-7492-4469-9707-be2a8509e1a0', 'installationId': '0673bfa5-1b1f-4f86-bfb2-9b8350b6c438', 'sessionId': 'e07b20e0-5658-41c0-8f52-eb49668fab2b', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.11', 'samcliVersion': '1.26.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'duration': 2856, 'exitReason': 'success', 'exitCode': 0}}]}
2021-07-26 23:06:57,058 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
{"Stacks":[...]}
2021-07-26 23:06:57,058 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-07-26 23:06:57,058 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId': 'aac28041-4f07-49c9-aad7-0b7a57fe9950', 'installationId': '0673bfa5-1b1f-4f86-bfb2-9b8350b6c438', 'sessionId': 'e07b20e0-5658-41c0-8f52-eb49668fab2b', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.11', 'samcliVersion': '1.26.0', 'runtimes': ['nodejs14.x']}}]}
2021-07-26 23:06:57,821 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: macOS 11.4
2. sam --version: SAM CLI, version 1.26.0
3. AWS region: us-east-1

Add --debug flag to command you are running

[twasink]

From examining the issue more, it’s almost certainly an issue with the lambci/lambda docker image. The scripts in the image kick off the mock lambda environment passing in environment variables, without checking if the environment is set or not. Thus, in the lambda’s runtime, the AWS_SESSION_TOKEN is set to a null or blank value.

In turn, the different lambda runtimes treat this differently. Some insect the value for truthiness, others look for the presence of the key.

So it ends up being a multi-layered bug: the SAM CLI initiates it by switching from whatever credentials the user has to using environment variables, the lambci/lambda image makes it worse by setting environment variables in the lambda runtime that weren’t passed to the Docker container, and the lambda runtime has inconsistent behaviour (across implementations) on how to deal with that.

[dtflowers]

We’ve also encountered this issue. This thread was quite informative and matches what we’re seeing exactly. Thank you for the help and +1 for the need to patch this bug.