issue with sockets

Environment details

• OS: Alpine (docker image https://hub.docker.com/_/python python:3.8.0-alpine)
• Python version: 3.8.0
• pip version: ???
• google-api-python-client version: v1.7.11

Steps to reproduce

1. Using Search Console API, make request for web property, that have millions of data.
2. Use the code snippet below. Where site_url is the URL of web property in Google Search Console that have enough data (100k+ data points).

from googleapiclient.discovery import build
service = build('webmasters', 'v3', credentials=credentials)
query = {'dimensions': ['date', 'page', 'query']}
query['startDate'] = '2019-01-01'
query['endDate'] = '2020-01-01'
query['rowLimit'] = 25000
query['startRow'] = 0
request = service.searchanalytics().query(siteUrl=site_url, body=query)
sometimes_response_sometimes_error = request.execute()

The error result appears irregularly, I was unable to find any additional conditions that makes this fail predictable.

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/rq/worker.py", line 886, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.8/site-packages/rq/job.py", line 664, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.8/site-packages/rq/job.py", line 670, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/usr/src/app/searchconsole/data.py", line 90, in load_data_for_property
    save_data_batch(property, batch)
  File "/usr/src/app/searchconsole/data.py", line 103, in save_data_batch
    for data_batch in data_batches:
  File "/usr/src/app/searchconsole/google/reports.py", line 103, in generate_lazy
    response = request.execute()
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 849, in execute
    resp, content = _retry_request(
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 184, in _retry_request
    raise exception
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 165, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/google_auth_httplib2.py", line 197, in request
    response, content = self.http.request(
  File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1982, in request
    (response, content) = self._request(
  File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1650, in _request
    (response, content) = self._conn_request(
  File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1589, in _conn_request
    response = conn.getresponse()
  File "/usr/local/lib/python3.8/http/client.py", line 1322, in getresponse
    response.begin()
  File "/usr/local/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.8/http/client.py", line 264, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

Quick fix?

I’m looking for just a quickfix if possible, reading other related issues, but changing httplib2 version doesn’t help.

[andrewkisel]

@vladignatyev As a quick fix, try increasing the default timeout of the socket. Note, do this right before this line: service = build('webmasters', 'v3', credentials=credentials). Example: socket.setdefaulttimeout(60 * 3). Can also try to pass num_retries=int to .execute(), not sure if it works in v1.7.11.

[busunkim96]

General socket timeout issue tracked in #709