Accept concurrent requests when using --warm-containers

Describe your idea/feature/enhancement

When developing serving an API locally you may run into concurrent requests done by a browser, and in macOS specifically without the --warm-containers option enabled it’s very hard to test an endpoint that renders HTML, for example (as each lambda invocation is going to take ~3s).

So, with that in mind, I’d love to see concurrent requests being supported when using one of the --warm-containers options.

By looking at the repository code it seems that lambda_runner.invoke is not expecting concurrent requests while reading the stdout stream which causes the concurrent requests to fail as the stdout is b'' and not a valid JSON response.

I came across this while searching about concurrency so, feel free to close this one if the team understands that this is cause more problems than a solution.

Proposal

I’d like to propose a way to accept concurrent requests in similar (couldn’t find a better word but, it’s far from similar) how NGINX and API Gateway deals with requests by accepting N of them but, since we have Docker as a limiting factor, processing it one at a time (similar to having a rate of 1 request/s but with 100 requests as a burst, for example).

I came up with this patch for it:

diff --git a/samcli/local/apigw/local_apigw_service.py b/samcli/local/apigw/local_apigw_service.py
index cc2684c2..93964c22 100644
--- a/samcli/local/apigw/local_apigw_service.py
+++ b/samcli/local/apigw/local_apigw_service.py
@@ -3,6 +3,7 @@ import io
 import json
 import logging
 import base64
+import threading
 from typing import List, Optional
 
 from flask import Flask, request
@@ -137,6 +138,7 @@ class LocalApigwService(BaseLocalService):
         self.static_dir = static_dir
         self._dict_of_routes = {}
         self.stderr = stderr
+        self._lock = threading.BoundedSemaphore()
 
     def create(self):
         """
@@ -155,6 +157,10 @@ class LocalApigwService(BaseLocalService):
         # Prevent the dev server from emitting headers that will make the browser cache response by default
         self._app.config["SEND_FILE_MAX_AGE_DEFAULT"] = 0
 
+        self._app.before_request(self._block_concurrent_requests)
+        self._app.teardown_request(self._release_request_lock)
+
+
         # This will normalize all endpoints and strip any trailing '/'
         self._app.url_map.strict_slashes = False
         default_route = None
@@ -256,6 +262,7 @@ class LocalApigwService(BaseLocalService):
         self._app.register_error_handler(405, ServiceErrorResponses.route_not_found)
         # Something went wrong
         self._app.register_error_handler(500, ServiceErrorResponses.lambda_failure_response)
+        self._app.register_error_handler(504, ServiceErrorResponses.gateway_timeout)
 
     def _request_handler(self, **kwargs):
         """
@@ -838,3 +845,16 @@ class LocalApigwService(BaseLocalService):
 
         """
         return request_mimetype in binary_types or "*/*" in binary_types
+
+    def _block_concurrent_requests(self):
+        # Use the hard timeout of APIG for integrations for each request.
+        acquired = self._lock.acquire(timeout=29)
+
+        if not acquired:
+            return ServiceErrorResponses.gateway_timeout()
+
+    def _release_request_lock(self, error=None):
+        try:
+            self._lock.release()
+        except ValueError:
+            LOG.debug("Tried to release more than it should.")
diff --git a/samcli/local/apigw/service_error_responses.py b/samcli/local/apigw/service_error_responses.py
index e2214dad..41231b9b 100644
--- a/samcli/local/apigw/service_error_responses.py
+++ b/samcli/local/apigw/service_error_responses.py
@@ -8,9 +8,11 @@ class ServiceErrorResponses:
     _NO_LAMBDA_INTEGRATION = {"message": "No function defined for resource method"}
     _MISSING_AUTHENTICATION = {"message": "Missing Authentication Token"}
     _LAMBDA_FAILURE = {"message": "Internal server error"}
+    _GATEWAY_TIMEOUT = {"message": "Gateway Timeout"}
 
     HTTP_STATUS_CODE_502 = 502
     HTTP_STATUS_CODE_403 = 403
+    HTTP_STATUS_CODE_504 = 504
 
     @staticmethod
     def lambda_failure_response(*args):
@@ -42,3 +44,8 @@ class ServiceErrorResponses:
         """
         response_data = jsonify(ServiceErrorResponses._MISSING_AUTHENTICATION)
         return make_response(response_data, ServiceErrorResponses.HTTP_STATUS_CODE_403)
+
+    @staticmethod
+    def gateway_timeout(*args):
+        response_data = jsonify(ServiceErrorResponses._GATEWAY_TIMEOUT)
+        return make_response(response_data, ServiceErrorResponses.HTTP_STATUS_CODE_504)

The code above uses a mutex to make the Flask threads wait until the previous one finishes by using Flask’s before_request and teardown_request. The first one tries to acquire the lock within a 29s timeout (which is the same used in APIG) while the second one releases it. If a thread can’t acquire the lock within the 29s interval it raises a 504 Bad gateway.

There’s a drawback to it, of course, if for some reason the lambda function times out the container doesn’t flush some of the requests (I didn’t look into it deeply but, it was what I could observe from my tests). It seems that maybe the invoke hangs? Or Docker hangs? Well, and this is the part where it may not be feasible or some guidance is needed to move forward.

Thanks for reading thru this!

[deldrid1]

This would be a HUGE boost to those of us doing local development using sam-cli (or sam-beta-cdk in my case).

Any updates on this patch coming in?

[jfuss]

Hey community,

I spoke with the team. When we designed out warm container support we intentionally did not support concurrent requests. For local emulation, we feel supporting reserved concurrency, warming pools, etc is out of scope. For testing these types of cases, we believe sam sync(https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/accelerate.html) is a better solution.

Closing.