Google Cloud Storage failing when using threads

1. Ubuntu 16.04
2. Python 2.7.6
3. google-api-python-client>=1.6.2 and google-cloud-storage>=1.1.1

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get
    raise self._value
ssl.SSLError: [Errno 1] _ssl.c:1429: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

1. The client is not thread safe (I think)

from multiprocessing.pool import ThreadPool
from google.cloud import storage
from functools import partial

def upload(bucket, i):
    blob = bucket.blob("file{}.png".format(i))
    blob.upload_from_string("blabla")
    blob.make_public()
    return blob.public_url

bucket = storage.Client().get_bucket("deepo-test")
pool = ThreadPool()
fct = partial(upload, bucket)
pool.map(fct, [i for i in range(2)])

[kumudraj]

you need to create a new client connection for every pool / thread inside def upload(bucket, i). That will be work.

from multiprocessing.pool import ThreadPool from google.cloud import storage

def upload(i): bucket = storage.Client().get_bucket(“deepo-test”) blob = bucket.blob(“file{}.png”.format(i)) blob.upload_from_string(“blabla”) blob.make_public() return blob.public_url

pool = ThreadPool() pool.map(fct, [i for i in range(2)])

[dhermes]

@Alexis-Jacob That’s correct, the error you are seeing is caused by the lack of thread-safety in httplib2. We recommend (for now) creating an instance of Client that is local to your thread / process.