Gevent + fork + google.cloud.storage + urllib3/requests gives "'module' object has no attribute 'epoll'"

Hello,

I am using the latest version of requests (and urllib3 1.22). I need to fork and monkey patch the child process (the parent has to stay unpatched). I also need to instantiate a google cloud storage client before forking and this gives me this traceback when downloading a page in the child:

Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gipc/gipc.py", line 361, in _child
    target(*args, **kwargs)
  File "debug_tests.py", line 17, in startProcess
    print requests.get("https://google.fr")
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 443, in wrap_socket
    rd = util.wait_for_read(sock, sock.gettimeout())
  File "/usr/local/lib/python2.7/dist-packages/urllib3/util/wait.py", line 33, in wait_for_read
    return _wait_for_io_events(socks, EVENT_READ, timeout)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/util/wait.py", line 22, in _wait_for_io_events
    with DefaultSelector() as selector:
  File "/usr/local/lib/python2.7/dist-packages/urllib3/util/selectors.py", line 581, in DefaultSelector
    return _DEFAULT_SELECTOR()
  File "/usr/local/lib/python2.7/dist-packages/urllib3/util/selectors.py", line 394, in __init__
    self._epoll = select.epoll()
AttributeError: 'module' object has no attribute 'epoll'

The smallest code I can give you is :

import os
import gipc
from google.cloud import storage

def start_process():
    from gevent import monkey
    monkey.patch_all()

    import requests
    print requests.get("https://google.fr")

class CloudStorage(object):

    def __init__(self, bucket_name):
        self.bucket_name = bucket_name
        storage_client = storage.Client()
        self.bucket = storage_client.get_bucket(self.bucket_name)


    def upload(self, remote_path):
        blob = self.bucket.blob(remote_path)
        blob.upload_from_filename("/tmp/test.txt")


if __name__ == "__main__":

    cloud_storage = CloudStorage("bucket-test")

    cloud_storage.upload("test")

    gipc.start_process(start_process)

I have tried :

• urllib3 1.19.1 with requests 2.18.4 works (I have a warning because the version of urllib is too old for requests, doesn’t work with higher versions of urllib3)
• urllib3 1.22 with requests 2.12.5 works (doesn’t work with higher versions of requests).
• Using gevent.fork or os.fork instead of gipc but it doesn’t change the error
• If I do patch_all(select=False) it works but without concurrency
• Uploading on s3 with boto3 instead works.
• Just doing a simple requests.get("https://google.fr") instead of creating a google_storage & uploading in the parent works. Note that if I only instantiate the google storage client, without uploading, it works.

For now I will go with the version 1.19.1 but it seems that something has been broken since ? Or is it in the requests module ?

Thank you very much,

[haikuginger]

Okay; I’m going to close this out, as I think we can agree that no action ought to be taken as a result of this issue.

[maingoh]

Just tried and it works ! I will stick with subprocess for now as it is safer 😉 Thanks !