Broken Pipe exceptions raised after idle periods

I’m using long-lived client objects like so:

client = discovery.build(....)
task = client.get_something(....).execute(num_retries=5)
some_long_running_process(task)
client.do_another_thing().execute(num_retries=5)

And running into exceptions on the calls to do_another_thing():

Traceback (most recent call last):
  File "/opt/app/gcp_helper.py", line 168, in get_size
    instanceGroupManager=instance_group).execute(num_retries=5)
  File "/opt/venv/local/lib/python2.7/site-packages/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/opt/venv/local/lib/python2.7/site-packages/googleapiclient/http.py", line 755, in execute
    method=str(self.method), body=self.body, headers=self.headers)
  File "/opt/venv/local/lib/python2.7/site-packages/googleapiclient/http.py", line 93, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/opt/venv/local/lib/python2.7/site-packages/oauth2client/client.py", line 633, in new_request
    self._refresh(request_orig)
  File "/opt/venv/local/lib/python2.7/site-packages/oauth2client/contrib/gce.py", line 129, in _refresh
    META, headers={'Metadata-Flavor': 'Google'})
  File "/opt/venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1609, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/opt/venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1351, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/opt/venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1273, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1001, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1035, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 850, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 826, in send
    self.sock.sendall(data)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe"

I would expect the num_retries argument to handle these exceptions in https://github.com/google/google-api-python-client/blob/master/googleapiclient/http.py#L145, but for the time being I have to implement my own retry logic around all API calls to overcome this.

[janneh]

I’m experiencing this issue as well…

Using:

Python 3.6.7

>>> import googleapiclient
>>> googleapiclient.__version__
'1.7.8'

Stack trace:

File "/var/task/pubsub_publisher.py", line 42, in handler
  pubsub.projects().topics().publish(topic=_GOOGLE_PUBSUB_TOPIC_NAME, body=body).execute() 
File "/var/task/googleapiclient/_helpers.py", line 130, in positional_wrapper
  return wrapped(*args, **kwargs)
File "/var/task/googleapiclient/http.py", line 846, in execute
  method=str(self.method), body=self.body, headers=self.headers)
File "/var/task/googleapiclient/http.py", line 183,in _retry_request
  raise exception
File "/var/task/googleapiclient/http.py", line 164, in _retry_request
  resp, content = http.request(uri, method, *args, **kwargs)
File "/var/task/google_auth_httplib2.py", line 198, in request
  uri, method, body=body, headers=request_headers, **kwargs)
File "/var/task/httplib2/__init__.py", line 1924, in request
  cachekey,
File "/var/task/httplib2/__init__.py", line 1595, in _request
  conn, request_uri, method, body, headers
File "/var/task/httplib2/__init__.py", line 1502, in _conn_request
  conn.request(method, request_uri, body, headers)
File "/var/lang/lib/python3.6/http/client.py", line 1239, in request
  self._send_request(method, url, body, headers, encode_chunked)
File "/var/lang/lib/python3.6/http/client.py", line 1285, in _send_request
  self.endheaders(body, encode_chunked=encode_chunked)
File "/var/lang/lib/python3.6/http/client.py", line 1234, in endheaders
  self._send_output(message_body, encode_chunked=encode_chunked)
File "/var/lang/lib/python3.6/http/client.py", line 1065, in _send_output
  self.send(chunk)
File "/var/lang/lib/python3.6/http/client.py", line 986, in send
  self.sock.sendall(data)
File "/var/lang/lib/python3.6/ssl.py", line 975, in sendall
  v = self.send(byte_view[count:])
File "/var/lang/lib/python3.6/ssl.py", line 944, in send
  return self._sslobj.write(data)
File "/var/lang/lib/python3.6/ssl.py", line 642, in write
  return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

It’s something that happens now and then, not super common. Constantly recurring though.

As an example in a sample of 7,741,334 publishes I have 6 of those errors.

[theacodes]

@Bobspadger probably not, but we’re soon replacing oauth2client in this library with google-auth, which will allow us to use urllib3 and requests without needing httplib2shim.