ConnectionResetError missing in retry

Here’s a callstack we just got and it wasn’t retried:

File "/pyenv/lib/python3.6/site-packages/googleapiclient/http.py", line 841, in execute
    method=str(self.method), body=self.body, headers=self.headers)
File "/pyenv/lib/python3.6/site-packages/googleapiclient/http.py", line 165, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
File "/pyenv/lib/python3.6/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
File "/pyenv/lib/python3.6/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
File "/pyenv/lib/python3.6/site-packages/httplib2/__init__.py", line 1514, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/pyenv/lib/python3.6/site-packages/httplib2/__init__.py", line 1264, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/pyenv/lib/python3.6/site-packages/httplib2/__init__.py", line 1217, in _conn_request
    response = conn.getresponse()
File "/pyenv/lib/python3.6/site-packages/aws_xray_sdk/core/recorder.py", line 374, in record_subsegment
    return_value = wrapped(*args, **kwargs)
File "/usr/local/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
File "/usr/local/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
File "/usr/local/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
File "/usr/local/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
File "/usr/local/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
File "/usr/local/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer

looks like a python3 support issue, this case is covered in py2 by socket.error. In my implementation I’ve centralized this logic here: https://github.com/thehesiod/google-api-python-client/blob/thehesiod/batch-retries/googleapiclient/http.py#L143

[thehesiod]

@eckardt <rant> the google APIs are IMHO a mess 😉 have a few issues. Here they switched to google-auth, however that doesn’t support token storage so depending on your impl you may do a lot of auth calls. Also beware of batched API calls as they didn’t add retries to that (nor cared for my fixes as technically this is just in “support” mode). Also there are some weird auth errors (saying you’re unauth’d when you are, fixed with retry, they didn’t want my fix for either. In the new cloud functions they switched to requests (https://github.com/GoogleCloudPlatform/google-cloud-python/pull/3674) as you’ve noticed, but there were a lot of fundamental bugs (https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4274, perhaps like you’re finding) so I’m still wary of using the new version. For now we’re on 0.26.0 of google-cloud-pubsub and related sub-modules. With my branch I’m able to run for ~10 days before a SSL leak in core python executable causes us to restart. </rant>

update: For httplib you’ll also need: https://github.com/httplib2/httplib2/pull/111, which isn’t released yet

[yan-hic]

Is there any traction on this ? ConnectionResetError has become very common for those using the api in Google Cloud Functions.

We raised this with Google Support and their answer was to implement a exponential backup. As per @thehesiod , adding the error as retriable would be an easy fix.