boto3 client times out (ReadTimeoutError) after synchronously invoking long running lambda even after lambda finishes

What issue did you see ?

I am seeing a weird issue with Lambda invocation when using boto3 client. Here is a brief summary:

boto3 client times out (ReadTimeoutError) after synchronously invoking long running lambda even after lambda finishes. This seems to only happen if the lambda function takes >350sec (even though the Lambda is configured with Timeout=600). I can confirm from AWS console and CW logs that the lambda finishes in ~350sec, but for some reason the boto3 client invocation times out after the boto3’s config read_timeout of 900sec. This doesn’t happen if the lambda runs <350sec.

Any help is greatly appreciated

Steps to reproduce

Handler code:

import logging
import time

log = logging.getLogger(__name__)


def my_handler(event, context):
    t1 = time.time()
    time.sleep(event.get("sleep_for", 1))
    response = {
        "I TOOK TIME": time.time() - t1,
    }
    log.debug(f"final response: {response}")
    return response

Invocation code:

import boto3
config = botocore.config.Config(
    read_timeout=900,
    connect_timeout=900,
    retries={"max_attempts": 0}
)

session = boto3.Session().client("lambda", config=config)
_client = session.client("lambda", config=config)
_client.invoke(
    FunctionName=fnc_name,
    Payload=json.dumps({"sleep_for": 350}}),
    InvocationType="RequestResponse",
)

• Using python3.7 (lambda handler and invocation code)
• CFN template to create lambda:

  MyLambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: handlers.my_lambda_handler
      Role:
        Fn::GetAtt: [MyLambdaFunctionRole, Arn]
      Timeout: 600            
      MemorySize: 2112        

Debug logs Full stack trace by adding boto3.set_stream_logger('') to your code.

Traceback (most recent call last):
  File "/myworkspace/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/myworkspace/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/myworkspace/python3.7/lib/python3.7/http/client.py", line 1344, in getresponse
    response.begin()
  File "/myworkspace/python3.7/lib/python3.7/http/client.py", line 306, in begin
    version, status, reason = self._read_status()
  File "/myworkspace/python3.7/lib/python3.7/http/client.py", line 267, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/myworkspace/python3.7/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/myworkspace/python3.7/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/myworkspace/python3.7/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/myworkspace/lib/python3.7/site-packages/botocore/httpsession.py", line 263, in send
    chunked=self._chunked(request.headers),
  File "/myworkspace/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/myworkspace/lib/python3.7/site-packages/urllib3/util/retry.py", line 343, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/myworkspace/lib/python3.7/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/myworkspace/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/myworkspace/lib/python3.7/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/myworkspace/lib/python3.7/site-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: AWSHTTPSConnectionPool(host='lambda.us-east-1.amazonaws.com', port=443): Read timed out. (read timeout=10)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/myworkspace/examples/run_integ_tsts_manually.py", line 86, in <module>
    run_from_ide()
  File "/myworkspace/examples/run_integ_tsts_manually.py", line 62, in run_from_ide
    fnc_input={"sleep_for": 350}))
  File "/lambda_helper.py", line 54, in invoke_lambda
    InvocationType="RequestResponse",
  File "/myworkspace/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/myworkspace/lib/python3.7/site-packages/botocore/client.py", line 613, in _make_api_call
    operation_model, request_dict, request_context)
  File "/myworkspace/lib/python3.7/site-packages/botocore/client.py", line 632, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/myworkspace/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/myworkspace/lib/python3.7/site-packages/botocore/endpoint.py", line 137, in _send_request
    success_response, exception):
  File "/myworkspace/lib/python3.7/site-packages/botocore/endpoint.py", line 231, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/myworkspace/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/myworkspace/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/myworkspace/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/myworkspace/lib/python3.7/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/myworkspace/lib/python3.7/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/myworkspace/lib/python3.7/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/myworkspace/lib/python3.7/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/myworkspace/lib/python3.7/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/myworkspace/lib/python3.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/myworkspace/lib/python3.7/site-packages/botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/myworkspace/lib/python3.7/site-packages/botocore/endpoint.py", line 244, in _send
    return self.http_session.send(request)
  File "/myworkspace/lib/python3.7/site-packages/botocore/httpsession.py", line 289, in send
    raise ReadTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/foo-HLO15WK0YX1W/invocations"

[swetashre]

@abarik1981 - Thank you for your post.

urllib3.exceptions.ReadTimeoutError: AWSHTTPSConnectionPool(host='lambda.us-east-1.amazonaws.com', port=443): Read timed out. (read timeout=10)

Here the read timeout should be 900 instead of 10 as you have specified the read timeout to 900 seconds. So please make sure you are using the correctly configured client.

Can you please try this code and provide me the output?

import boto3
import time

config = botocore.config.Config(
    read_timeout=900,
    connect_timeout=900,
    retries={"max_attempts": 0}
)

start = time.time()
boto3.set_stream_logger('')
session = boto3.Session()
client = session.client("lambda", config=config)

client.invoke(
    FunctionName=fnc_name,
    Payload=json.dumps({"sleep_for": 350}}),
    InvocationType="RequestResponse",
)
end = time.time()
print("total time is {}".format(end-start))

Please provide me your cloudwatch logs as well after running this code. This is the troubleshooting documentation that might help: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-retry-timeout-sdk/

[ArielPrevu3D]

Lambda PrivateLink endpoints will give me an idle connection 1% to 5% of the time, regardless of VPC settings of your function. The invocation makes it to the function, but then the socket gets ReadTimeout. The issue doesn’t exist when your instance accesses Lambda through an internet gateway or when testing from my computer. That’s very strange, since PrivateLink is supposed to help with availability, reliability and scaling.