Bug: The security token included in the request is invalid

Description:

I did find a similar issue but was closed and despite new comments added is not re-opened.

Invoking an image runtime lambda locally produces the error in the title when accessing Secrets Manager.

Steps to reproduce:

Create sample dockerized Lambda, try accessing Secrets Manager.

Invoke: sam local invoke "Function" -e events/test_sm.json --profile *** --debug

Observed result:

2022-06-14 22:42:44,359 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2022-06-14 22:42:44,359 | Using config file: samconfig.toml, config environment: default
2022-06-14 22:42:44,359 | Expand command line arguments to:
2022-06-14 22:42:44,359 | --template_file=/home/***/code/***/***-***/.aws-sam/build/template.yaml --event=events/test_sm.json --function_logical_id=Function --no_event --layer_cache_basedir=/home/***/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1 
2022-06-14 22:42:44,359 | local invoke command is called
2022-06-14 22:42:44,363 | Collected default values for parameters: {}
2022-06-14 22:42:44,371 | Sam customer defined id is more priority than other IDs. Customer defined id for resource Function is Function
2022-06-14 22:42:44,371 | 0 stacks found in the template
2022-06-14 22:42:44,371 | Collected default values for parameters: {}
2022-06-14 22:42:44,378 | Sam customer defined id is more priority than other IDs. Customer defined id for resource Function is Function
2022-06-14 22:42:44,378 | 1 resources found in the stack 
2022-06-14 22:42:44,379 | Found Serverless function with name='Function' and ImageUri='function:latest'
2022-06-14 22:42:44,379 | --base-dir is not presented, adjusting uri /home/***/code/***/***-*** relative to /home/***/code/***/***-***/.aws-sam/build/template.yaml
2022-06-14 22:42:44,379 | --base-dir is not presented, adjusting uri . relative to /home/***/code/***/***-***/.aws-sam/build/template.yaml
2022-06-14 22:42:44,382 | Found one Lambda function with name 'Function'
2022-06-14 22:42:44,382 | Invoking Container created from function:latest
2022-06-14 22:42:44,382 | Environment variables overrides data is standard format
2022-06-14 22:42:44,382 | Loading AWS credentials from session with profile '***'
2022-06-14 22:42:44,389 | Code None is not a zip/jar file
Building image.................
2022-06-14 22:42:44,463 | Skip pulling image and use local one: function:rapid-1.50.0-x86_64.

2022-06-14 22:42:44,647 | Starting a timer for 300 seconds for function 'Function'
START RequestId: 27b5c17f-7b42-4dcf-85d2-3df84810bea1 Version: $LATEST
Skipped bootstraping TelemetryLog
Executing 'lambda.lambda_handler' in function directory '/function'
Error raised from handler method
{
  "errorMessage": "The security token included in the request is invalid.",
  "errorType": "Function<Aws::SecretsManager::Errors::UnrecognizedClientException>",
  "stackTrace": [
    "/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'",
    "/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'",
    "/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'",
    "/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'",
    "/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'",
    "/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/seahorse/client/plugins/request_callback.rb:71:in `call'",
    "/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'",
    "/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/seahorse/client/plugins/response_target.rb:24:in `call'",
    "/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/seahorse/client/request.rb:72:in `send_request'",
    "/usr/local/bundle/gems/aws-sdk-secretsmanager-1.62.0/lib/aws-sdk-secretsmanager/client.rb:1323:in `get_secret_value'",
    "/function/secrets_manager.rb:39:in `read_secret_json'",
    "/function/secrets_manager.rb:15:in `initialize'",
    "/function/lambda.rb:15:in `new'",
    "/function/lambda.rb:15:in `initialize'",
    "/function/lambda.rb:46:in `new'",
    "/function/lambda.rb:46:in `lambda_handler'",
    "/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric/lambda_handler.rb:28:in `call_handler'",
    "/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric.rb:81:in `run_user_code'",
    "/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric.rb:59:in `start_runtime_loop'",
    "/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric.rb:42:in `run'",
    "/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric/bootstrap.rb:35:in `bootstrap_handler'",
    "/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric/bootstrap.rb:8:in `start'",
    "/usr/local/bundle/gems/aws_lambda_ric-2.0.0/bin/aws_lambda_ric:10:in `<top (required)>'",
    "/usr/local/bundle/bin/aws_lambda_ric:25:in `load'",
    "/usr/local/bundle/bin/aws_lambda_ric:25:in `<main>'"
  ]
}
END RequestId: 27b5c17f-7b42-4dcf-85d2-3df84810bea1
REPORT RequestId: 27b5c17f-7b42-4dcf-85d2-3df84810bea1	Init Duration: 0.11 ms	Duration: 1068.71 ms	Billed Duration: 1069 ms	Memory Size: 3072 MB	Max Memory Used: 3072 MB	
{"errorMessage":"The security token included in the request is invalid.","errorType":"Function<Aws::SecretsManager::Errors::UnrecognizedClientException>","stackTrace":["/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'","/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'","/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'","/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'","/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'","/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/seahorse/client/plugins/request_callback.rb:71:in `call'","/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'","/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/seahorse/client/plugins/response_target.rb:24:in `call'","/usr/local/bundle/gems/aws-sdk-core-3.131.1/lib/seahorse/client/request.rb:72:in `send_request'","/usr/local/bundle/gems/aws-sdk-secretsmanager-1.62.0/lib/aws-sdk-secretsmanager/client.rb:1323:in `get_secret_value'","/function/secrets_manager.rb:39:in `read_secret_json'","/function/secrets_manager.rb:15:in `initialize'","/function/lambda.rb:15:in `new'","/function/lambda.rb:15:in `initialize'","/function/lambda.rb:46:in `new'","/function/lambda.rb:46:in `lambda_handler'","/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric/lambda_handler.rb:28:in `call_handler'","/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric.rb:81:in `run_user_code'","/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric.rb:59:in `start_runtime_loop'","/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric.rb:42:in `run'","/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric/bootstrap.rb:35:in `bootstrap_handler'","/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric/bootstrap.rb:8:in `start'","/usr/local/bundle/gems/aws_lambda_ric-2.0.0/bin/aws_lambda_ric:10:in `<top (required)>'","/usr/local/bundle/bin/aws_lambda_ric:25:in `load'","/usr/local/bundle/bin/aws_lambda_ric:25:in `<main>'"]}2022-06-14 22:42:45,880 | Cleaning all decompressed code dirs
2022-06-14 22:42:45,881 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '30b28989-9feb-4a2f-870f-a7c09a293b99', 'installationId': '0b0f3b2b-3efb-438e-b497-cf6bf622fdf6', 'sessionId': 'fbf696f2-5466-43d2-bbc7-8505ad005ea3', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.7.10', 'samcliVersion': '1.50.0', 'awsProfileProvided': True, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN'}, 'duration': 1521, 'exitReason': 'success', 'exitCode': 0}}]}
2022-06-14 22:42:46,510 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
2022-06-14 22:42:46,511 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2022-06-14 22:42:46,511 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId': '83b04abb-a537-489d-a6e7-c4ba9d515844', 'installationId': '0b0f3b2b-3efb-438e-b497-cf6bf622fdf6', 'sessionId': 'fbf696f2-5466-43d2-bbc7-8505ad005ea3', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.7.10', 'samcliVersion': '1.50.0', 'runtimes': [None]}}]}
2022-06-14 22:42:47,130 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

Expected result:

Response from my function

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Linux
2. sam --version: SAM CLI, version 1.50.0
3. AWS region: eu-west-2

[darioackermann]

Experiencing the same issue. Have you been able to resolve it?

[sriram-mv]

Thanks for documenting your journey! I’m going to close this particular issue. But this issue will still be searchable for other folks who might run into this.