loky crash when used from a uwsgi process

When issuing a HTTP request on a uswgi app that uses loky one get the following error messages:

unable to load configuration from from loky.backend.semaphore_tracker import main; main(10)
Fri Apr 19 16:39:13 2019 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request / (ip 127.0.0.1) !!!
/home/ogrisel/miniconda3/envs/pip/lib/python3.7/site-packages/loky/backend/semaphore_tracker.py:74: UserWarning: semaphore_tracker: process died unexpectedly, relaunching.  Some semaphores might leak.
  warnings.warn('semaphore_tracker: process died unexpectedly, '
/home/ogrisel/miniconda3/envs/pip/bin/uwsgi: unrecognized option '--process-name'
getopt_long() error
/home/ogrisel/miniconda3/envs/pip/bin/uwsgi: unrecognized option '--process-name'
getopt_long() error
/home/ogrisel/miniconda3/envs/pip/bin/uwsgi: unrecognized option '--process-name'
getopt_long() error
unable to load configuration from from loky.backend.semaphore_tracker import main; main(11)
/home/ogrisel/miniconda3/envs/pip/bin/uwsgi: unrecognized option '--process-name'
getopt_long() error

and then:

Traceback (most recent call last):
/home/ogrisel/miniconda3/envs/pip/bin/uwsgi: unrecognized option '--process-name'
getopt_long() error
  File "loky_wsgi_app.py", line 8, in application
    result = e.submit(lambda: 42).result()
  File "/home/ogrisel/miniconda3/envs/pip/lib/python3.7/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/home/ogrisel/miniconda3/envs/pip/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker. The exit codes of the workers are {EXIT(1), EXIT(1), EXIT(1)}

Steps to reproduce

• Install uswgi: for some reason I could not compile it with recent gcc versions, but it works with clang, e.g.:

CC=clang-7 pip install uswgi

• write a small uswgi app that uses loky, for instance a file named “loky_wsgi_app.py” with the following content:

from loky import get_reusable_executor


def application(env, start_response):
    e = get_reusable_executor()
    result = e.submit(lambda: 42).result()

    start_response('200 OK', [('Content-Type','text/html')])
    return [b"%d" % result]

Start the server:

uwsgi --http :9999 --wsgi-file loky_wsgi_app.py

and navigate to http://localhost:9999 .

[ogrisel]

To fix this we basically need to find a python executable in the parent folder of the uwsgi executable pointed by sys.executable that can import loky from the same installation path (under that is under site-packages folder).

If several Python versions are installed this might be a bit expensive to do so this should be cached.

[ogrisel]

If someone wants to volunteer to implement the strategy described here: https://github.com/joblib/loky/issues/207#issuecomment-485137208