Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Using --warm-containers EAGER option doesn't allow concurrent lambda execution
See original GitHub issueDescription:
When starting local lambda server with container reuse enabled sending a request before the previous one completes causes both to fail.
Steps to reproduce:
This behavior can be observed by changing container_mode in the existing integration tests:
index daed2ca..5fd5f7c 100644
--- a/tests/integration/local/start_lambda/test_start_lambda.py
+++ b/tests/integration/local/start_lambda/test_start_lambda.py
@@ -19,6 +19,7 @@ from .start_lambda_api_integ_base import StartLambdaIntegBaseClass
class TestParallelRequests(StartLambdaIntegBaseClass):
template_path = "/testdata/invoke/template.yml"
+ container_mode = ContainersInitializationMode.EAGER.value
def setUp(self):
self.url = "http://127.0.0.1:{}".format(self.port)
This test passes when using LAZY option because in that case container reuse won’t happen. Sending a “warm-up” requests before will cause the test to fail in a similar fashion.
Observed result:
I did not find any useful information in the debug log, however, when running in LAZY mode with warm-up requests aws-lambda-rie prints the following error message:
time="2020-12-23T16:39:25.688" level=error msg="Failed to reserve: AlreadyReserved"
Full log for one attempt:
-------------------------------------------------------------------------------------------------------- Captured stderr setup --------------------------------------------------------------------------------------------------------
Starting the Local Lambda Service. You can now invoke your Lambda Functions defined in your template through the endpoint.
2020-12-23 17:46:08 * Running on http://127.0.0.1:30012/ (Press CTRL+C to quit)
-------------------------------------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------------------------------------
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.6:rapid-1.15.0.
Mounting /aws-sam-cli/tests/integration/testdata/invoke as /var/task:ro,delegated inside runtime container
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.6:rapid-1.15.0.
Mounting /aws-sam-cli/tests/integration/testdata/invoke as /var/task:ro,delegated inside runtime container
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.6:rapid-1.15.0.
Mounting /aws-sam-cli/tests/integration/testdata/invoke as /var/task:ro,delegated inside runtime container
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.6:rapid-1.15.0.
Mounting /aws-sam-cli/tests/integration/testdata/invoke as /var/task:ro,delegated inside runtime container
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.6:rapid-1.15.0.
Mounting /aws-sam-cli/tests/integration/testdata/invoke as /var/task:ro,delegated inside runtime container
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.6:rapid-1.15.0.
Mounting /aws-sam-cli/tests/integration/testdata/invoke as /var/task:ro,delegated inside runtime container
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.6:rapid-1.15.0.
Mounting /aws-sam-cli/tests/integration/testdata/invoke as /var/task:ro,delegated inside runtime container
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.6:rapid-1.15.0.
Mounting /aws-sam-cli/tests/integration/testdata/invoke as /var/task:ro,delegated inside runtime container
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.6:rapid-1.15.0.
Mounting /aws-sam-cli/tests/integration/testdata/invoke as /var/task:ro,delegated inside runtime container
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.6:rapid-1.15.0.
Mounting /aws-sam-cli/tests/integration/testdata/invoke as /var/task:ro,delegated inside runtime container
START RequestId: 6e05694f-d843-4dd8-9c57-77013dd62661 Version: $LATEST
START RequestId: 2c060a6f-5bac-4765-9452-13a82140cf79 Version: $LATEST
Loading function
START RequestId: 0daa0978-68ba-4860-9225-3822226fe2f5 Version: $LATEST
START RequestId: 0971a594-6b30-4fd9-87ec-5926c3958017 Version: $LATEST
Loading function
Loading function
Loading function
START RequestId: b29e5112-112e-4174-bc61-c6e3c2522836 Version: $LATEST
START RequestId: 1a2d6829-afd4-4d5d-a6ad-9668ae1cc836 Version: $LATEST
START RequestId: 647a89d6-b482-412c-b0ea-df2a6b20d2b7 Version: $LATEST
Loading function
Loading function
START RequestId: db6a8512-8320-4d74-bcf5-266f24f99a18 Version: $LATEST
Loading function
Loading function
START RequestId: 4982f472-f18a-46cb-8b10-574d12f70811 Version: $LATEST
START RequestId: dc025608-4506-4800-a4ca-d6bfe2df78df Version: $LATEST
Loading function
Loading function
END RequestId: 6e05694f-d843-4dd8-9c57-77013dd62661
REPORT RequestId: 6e05694f-d843-4dd8-9c57-77013dd62661 Init Duration: 0.58 ms Duration: 10596.13 ms Billed Duration: 10600 ms Memory Size: 128 MB Max Memory Used: 128 MB
2020-12-23 17:46:28 127.0.0.1 - - [23/Dec/2020 17:46:28] "POST /2015-03-31/functions/HelloWorldSleepFunction/invocations HTTP/1.1" 200 -
END RequestId: 2c060a6f-5bac-4765-9452-13a82140cf79
REPORT RequestId: 2c060a6f-5bac-4765-9452-13a82140cf79 Init Duration: 0.37 ms Duration: 10615.47 ms Billed Duration: 10700 ms Memory Size: 128 MB Max Memory Used: 128 MB
2020-12-23 17:46:28 127.0.0.1 - - [23/Dec/2020 17:46:28] "POST /2015-03-31/functions/HelloWorldSleepFunction/invocations HTTP/1.1" 200 -
END RequestId: 0daa0978-68ba-4860-9225-3822226fe2f5
REPORT RequestId: 0daa0978-68ba-4860-9225-3822226fe2f5 Init Duration: 1.29 ms Duration: 10682.40 ms Billed Duration: 10700 ms Memory Size: 128 MB Max Memory Used: 128 MB
2020-12-23 17:46:29 127.0.0.1 - - [23/Dec/2020 17:46:29] "POST /2015-03-31/functions/HelloWorldSleepFunction/invocations HTTP/1.1" 200 -
END RequestId: 0971a594-6b30-4fd9-87ec-5926c3958017
REPORT RequestId: 0971a594-6b30-4fd9-87ec-5926c3958017 Init Duration: 9.90 ms Duration: 10567.74 ms Billed Duration: 10600 ms Memory Size: 128 MB Max Memory Used: 128 MB
2020-12-23 17:46:29 127.0.0.1 - - [23/Dec/2020 17:46:29] "POST /2015-03-31/functions/HelloWorldSleepFunction/invocations HTTP/1.1" 200 -
END RequestId: b29e5112-112e-4174-bc61-c6e3c2522836
REPORT RequestId: b29e5112-112e-4174-bc61-c6e3c2522836 Init Duration: 0.39 ms Duration: 10582.72 ms Billed Duration: 10600 ms Memory Size: 128 MB Max Memory Used: 128 MB
2020-12-23 17:46:29 127.0.0.1 - - [23/Dec/2020 17:46:29] "POST /2015-03-31/functions/HelloWorldSleepFunction/invocations HTTP/1.1" 200 -
2020-12-23 17:46:30 127.0.0.1 - - [23/Dec/2020 17:46:30] "POST /2015-03-31/functions/HelloWorldSleepFunction/invocations HTTP/1.1" 200 -
END RequestId: 1a2d6829-afd4-4d5d-a6ad-9668ae1cc836
REPORT RequestId: 1a2d6829-afd4-4d5d-a6ad-9668ae1cc836 Init Duration: 1.90 ms Duration: 10377.85 ms Billed Duration: 10400 ms Memory Size: 128 MB Max Memory Used: 128 MB
2020-12-23 17:46:30 127.0.0.1 - - [23/Dec/2020 17:46:30] "POST /2015-03-31/functions/HelloWorldSleepFunction/invocations HTTP/1.1" 200 -
END RequestId: 647a89d6-b482-412c-b0ea-df2a6b20d2b7
REPORT RequestId: 647a89d6-b482-412c-b0ea-df2a6b20d2b7 Init Duration: 0.24 ms Duration: 10369.04 ms Billed Duration: 10400 ms Memory Size: 128 MB Max Memory Used: 128 MB
2020-12-23 17:46:30 127.0.0.1 - - [23/Dec/2020 17:46:30] "POST /2015-03-31/functions/HelloWorldSleepFunction/invocations HTTP/1.1" 200 -
END RequestId: db6a8512-8320-4d74-bcf5-266f24f99a18
REPORT RequestId: db6a8512-8320-4d74-bcf5-266f24f99a18 Init Duration: 0.31 ms Duration: 10199.79 ms Billed Duration: 10200 ms Memory Size: 128 MB Max Memory Used: 128 MB
END RequestId: 4982f472-f18a-46cb-8b10-574d12f70811
REPORT RequestId: 4982f472-f18a-46cb-8b10-574d12f70811 Init Duration: 0.42 ms Duration: 10250.64 ms Billed Duration: 10300 ms Memory Size: 128 MB Max Memory Used: 128 MB
2020-12-23 17:46:30 127.0.0.1 - - [23/Dec/2020 17:46:30] "POST /2015-03-31/functions/HelloWorldSleepFunction/invocations HTTP/1.1" 200 -
2020-12-23 17:46:30 127.0.0.1 - - [23/Dec/2020 17:46:30] "POST /2015-03-31/functions/HelloWorldSleepFunction/invocations HTTP/1.1" 200 -
END RequestId: dc025608-4506-4800-a4ca-d6bfe2df78df
REPORT RequestId: dc025608-4506-4800-a4ca-d6bfe2df78df Init Duration: 0.67 ms Duration: 10233.64 ms Billed Duration: 10300 ms Memory Size: 128 MB Max Memory Used: 128 MB
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Invoking main.sleep_handler (python3.6)
Reuse the created warm container for Lambda function 'HelloWorldSleepFunction'
Reuse the created warm container for Lambda function 'HelloWorldSleepFunction'
Reuse the created warm container for Lambda function 'HelloWorldSleepFunction'
Reuse the created warm container for Lambda function 'HelloWorldSleepFunction'
Lambda function 'HelloWorldSleepFunction' is already running
Reuse the created warm container for Lambda function 'HelloWorldSleepFunction'
Reuse the created warm container for Lambda function 'HelloWorldSleepFunction'
Reuse the created warm container for Lambda function 'HelloWorldSleepFunction'
Reuse the created warm container for Lambda function 'HelloWorldSleepFunction'
Reuse the created warm container for Lambda function 'HelloWorldSleepFunction'
Lambda function 'HelloWorldSleepFunction' is already running
Lambda function 'HelloWorldSleepFunction' is already running
Lambda function 'HelloWorldSleepFunction' is already running
Reuse the created warm container for Lambda function 'HelloWorldSleepFunction'
Lambda function 'HelloWorldSleepFunction' is already running
Lambda function 'HelloWorldSleepFunction' is already running
Lambda function 'HelloWorldSleepFunction' is already running
Lambda function 'HelloWorldSleepFunction' is already running
START RequestId: 8fbfeb0f-59fa-4aeb-9ea2-ed3e7441e426 Version: $LATEST
Lambda function 'HelloWorldSleepFunction' is already running
Lambda function 'HelloWorldSleepFunction' is already running
2020-12-23 17:46:31 127.0.0.1 - - [23/Dec/2020 17:46:31] "POST /2015-03-31/functions/HelloWorldSleepFunction/invocations HTTP/1.1" 200 -
======================================================================================================= short test summary info =======================================================================================================
FAILED tests/integration/local/start_lambda/test_start_lambda.py::TestParallelRequests::test_same_endpoint - botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "http://127.0.0.1:30012/2015-03-31/functions/HelloW...
======================================================================================== 1 failed, 18 deselected, 3 rerun in 570.85s (0:09:30) ========================================================================================
Expected result:
Reusing containers shouldn’t break lambda concurrency.
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
- OS: Mac
sam --version: SAM CLI, version 1.15.0 (master)
Issue Analytics
- State:
- Created 3 years ago
- Reactions:4
- Comments:7 (1 by maintainers)
Top Results From Across the Web
sam local start-api - AWS Serverless Application Model
Specifies the Lambda function to apply debug options to when --warm-containers is specified. This parameter applies to --debug-port , --debugger-path ...
Read more >AWS Lambda - [ERROR] (rapid) Failed to reserve
I'm using a containerized Lambda deployment (via Docker). The flow is: There's a Monday.com board (just like an excel spreadsheet) where some ...
Read more >AWS Lambda Functions Global State | by Brian Olson
This gives us the option to have sam local maintain the container through invocations. It turns out either option ( eager and lazy...
Read more >Eric Johnson on Twitter: "Thaaat's right! AWS SAM CLI 1.14.0 ...
Using container image support for AWS Lambda with AWS SAM | Amazon Web Services ... with "sam local start-api --warm-containers eager|lazy".
Read more >Cold Starts - Lamby: Simple Rails & AWS Lambda Integration
AWS provides an option called Provisioned Concurrency (PC) which allows you to warm instances prior to receiving requests. This lets you execute Lambda...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@mntss Thanks for the issue. Currently the EAGER mode does not support multiple requests and is by design. We should have a better way to handle this, but I am not aware of any hooks we have into the function that would allow that.
SAM CLI’s “warm” containers is not the same as Lambda Concurrency and should not be viewed as such. Warm containers, means we will not tear does the container after every invoke locally. This helps speed up the local testing cycles, not to replicate Lambda’s warming of functions mechanism. We cannot provide the same warming of functions as Lambda does locally, since there is no way for us to know if the container is up and running waiting for a request or processing one (among other differences in freeze/thaw, etc). So if you are sending multiple requests, you must use a mode that is not going to re-use a running container.
Not being able to use
--warm-containerswith concurrent requests is bad news. Any plans to solve this somehow?