Connection refused, ending in timeout lambda

Describe what happened: We’re monitoring all lambda errors in our account, and thus we monitor the datadog lambda (version 1.5.0) as well. We see reoccurring failures i.e. at least once a day.

Looking through the cloudwatch logs, this is what I see:

[Errno 111] Connection refused: error
Traceback (most recent call last):
File "/opt/python/lib/python2.7/site-packages/datadog_lambda/wrapper.py", line 58, in __call__
return self.func(event, context)
File "/var/task/lambda_function.py", line 380, in datadog_forwarder
forward_logs(filter_logs(logs))
File "/var/task/lambda_function.py", line 402, in forward_logs
with DatadogClient(cli) as client:
File "/var/task/lambda_function.py", line 193, in __enter__
self._client.__enter__()
File "/var/task/lambda_function.py", line 242, in __enter__
self._connect()
File "/var/task/lambda_function.py", line 215, in _connect
sock.connect((self.host, self.port))
File "/usr/lib64/python2.7/ssl.py", line 864, in connect
self._real_connect(addr, False)
File "/usr/lib64/python2.7/ssl.py", line 851, in _real_connect
socket.connect(self, addr)
File "/usr/lib64/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused

Followed by a REPORT log entry saying the lambda consumed all the allocated time.

Describe what you expected: I expect the lambda not to fail. Not at such a high rate anyway.

Steps to reproduce the issue:

Given the high failure rate, I presume this is smth that happens to everyone.

[cbanciu667]

Hi @andreineculau I struggled for some day with the same error until i switched to Python3.6 and latest lambda layer version in my region (arn:aws:lambda:eu-central-1:464622532012:layer:Datadog-Python36:5). After that everything worked as expected.

[tianchu]

@andreineculau I hope you have resolved the issue with our support team, and I’m going to close this issue. Feel free to reopen if you still need help from us.