httplib2.Http is not thread-safe

This is from a StackOverflow post. I’ve done some debugging from the Datastore side but I don’t think these requests ever make it into the Datastore part of the stack. I’d appreciate if someone look at this from the gcloud-python team. Note that this issue was originally reported via [e-mail] In July.

I have a Python Django application running on a Google Compute instance. It is using gcloudoem to interface from Django to Google Datastore. gcloudoem uses the same underlying code to communicate with Datastore as gcloud-python 0.5.x

At what seems to be completely random times, I will get SSL errors happening when trying to talk to Datastore. There is no pattern in where in my application code these happen. It’s just during a random call to Datastore. Here are the two flavours of errors:

ERROR:django.request:Internal Server Error: /complete/google-oauth2/
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/cache.py", line 52, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/csrf.py", line 57, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/apps/django_app/utils.py", line 51, in wrapper
    return func(request, backend, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/apps/django_app/views.py", line 28, in complete
    redirect_name=REDIRECT_FIELD_NAME, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/actions.py", line 43, in do_complete
    user = backend.complete(user=user, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/base.py", line 41, in complete
    return self.auth_complete(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/utils.py", line 229, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/oauth.py", line 387, in auth_complete
    *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/utils.py", line 229, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/oauth.py", line 396, in do_auth
    return self.strategy.authenticate(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/strategies/django_strategy.py", line 96, in authenticate
    return authenticate(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/__init__.py", line 60, in authenticate
    user = backend.authenticate(**credentials)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/base.py", line 82, in authenticate
    return self.pipeline(pipeline, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/base.py", line 85, in pipeline
    out = self.run_pipeline(pipeline, pipeline_index, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/base.py", line 112, in run_pipeline
    result = func(*args, **out) or {}
  File "/usr/local/lib/python2.7/dist-packages/social/pipeline/social_auth.py", line 20, in social_user
    social = backend.strategy.storage.user.get_social_auth(provider, uid)
  File "./social_gc/storage.py", line 105, in get_social_auth
    return cls.objects.get(provider=provider, uid=uid)
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/queryset/__init__.py", line 162, in get
    num = len(clone)
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/queryset/__init__.py", line 126, in __len__
    self._fetch_all()
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/queryset/__init__.py", line 370, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/query.py", line 480, in __iter__
    self.next_page()
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/query.py", line 452, in next_page
    transaction_id=transaction and transaction.id,
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/connection.py", line 249, in run_query
    response = self._rpc('runQuery', request, datastore_pb.RunQueryResponse)
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/connection.py", line 159, in _rpc
    data=request_pb.SerializeToString()
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/connection.py", line 134, in _request
    body=data
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 589, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1609, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1351, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1307, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1127, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 453, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 734, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 621, in read
    v = self._sslobj.read(len or 1024)
SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1752)

Unfortunately, for the second, I don’t have a full stacktrace handy:

[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:1752)

These errors don’t happen when I am using the GCD tool. Does anyone have any idea what is happening here? Is this some sort of networking problem?

[dhermes]

For those still following this issue, #3674 is putting the final nail in our usage of httplib2

[sadovnychyi]

As a fast workaround I wrote a custom wrapper around requests to pretend to be httplib2.

class CustomHttp(object):
  def __init__(self, timeout=None):
    self.timeout = timeout

  def request(self, uri, method='GET', body=None, headers=None,
              redirections=None, connection_type=None):
    if connection_type is not None:
      uri = '%s://%s' % (connection_type, uri)
    resp = requests.request(method=method, url=uri, data=body, headers=headers,
                            timeout=self.timeout)
    resp.status = resp.status_code
    return resp, resp.content

credentials.authorize(CustomHttp())

Works great now. Maybe useful to somebody.