Google Drive Batch Request Error: socket.timeout: The read operation timed out

Possibly related to #563

Occasionally when executing a batch request using the Python client library callingdrive.files().copy the operation fails with the error socket.timeout: The read operation timed out

Environment details

Running Python 3.7 on Google Cloud Functions environment

Relevant Code

def copy_files(access_token, file_ids, folder_ids, labels):
    def copy_callback(request_id, response, exception):
        if exception:
            print(exception)
        else:
            print(response)

    creds = AccessTokenCredentials(access_token, 'my-user-agent/1.0')
    drive_service = build('drive', 'v3', credentials=creds)

    move_files = drive_service.new_batch_http_request(callback=copy_callback)

    for label, id in zip(labels, file_ids):
        move_files.add(
            drive_service.files().copy(fileId=id, body={"parents": [folder_ids[label]]})
        )

    move_files.execute()

Error

Traceback (most recent call last):
  File "/env/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 313, in run_http_function
    result = _function_handler.invoke_user_function(flask.request)
  File "/env/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 215, in invoke_user_function
    return call_user_function(request_or_event)
  File "/env/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 208, in call_user_function
    return self._user_function(request_or_event)
  File "/user_code/main.py", line 19, in sort
    request_json['labels'])
  File "/user_code/drive.py", line 51, in copy_files
    move_files.execute()
  File "/env/lib/python3.7/site-packages/google_api_python_client-1.7.8-py3.7.egg/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/env/lib/python3.7/site-packages/google_api_python_client-1.7.8-py3.7.egg/googleapiclient/http.py", line 1450, in execute
    self._execute(http, self._order, self._requests)
  File "/env/lib/python3.7/site-packages/google_api_python_client-1.7.8-py3.7.egg/googleapiclient/http.py", line 1382, in _execute
    headers=headers)
  File "/env/lib/python3.7/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
  File "/env/lib/python3.7/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
  File "/env/lib/python3.7/site-packages/httplib2/__init__.py", line 1926, in request
    cachekey,
  File "/env/lib/python3.7/site-packages/httplib2/__init__.py", line 1595, in _request
    conn, request_uri, method, body, headers
  File "/env/lib/python3.7/site-packages/httplib2/__init__.py", line 1533, in _conn_request
    response = conn.getresponse()
  File "/opt/python3.7/lib/python3.7/http/client.py", line 1321, in getresponse
    response.begin()
  File "/opt/python3.7/lib/python3.7/http/client.py", line 296, in begin
    version, status, reason = self._read_status()
  File "/opt/python3.7/lib/python3.7/http/client.py", line 257, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/opt/python3.7/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/opt/python3.7/lib/python3.7/ssl.py", line 1052, in recv_into
    return self.read(nbytes, buffer)
  File "/opt/python3.7/lib/python3.7/ssl.py", line 911, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

Thanks!

[busunkim96]

Thank you @AlJohri for linking that issue! 😃

The default timeout 60 seconds, which is probably too short for many batch requests. It makes sense that the behavior would go away when batches were made smaller (smaller batches -> faster requests).

https://github.com/googleapis/google-api-python-client/blob/f0b025b95b2fb5fd0369d7233614e35b9c86af30/googleapiclient/http.py#L1777-L1792

import socket
from googleapiclient import discovery

socket.setdefaulttimeout(600)  # set timeout to 10 minutes
client = discovery.build("drive", "v3")

I am closing this issue now, but please do open a new issue if you are still experiencing problems.

[ktbernoulli]

Hiya, I have been seeing this error when I run reports for Google Analytics, a lot during 11am to 3pm PDT yesterday & just today again so far at 3am PDT. Is there a solution to this so far? I also updated to the 1.7.9 version on all forms.