Parallel class does not use temporary directory

I do not have /dev/shm on the instance I’m using (an AWS lambda) and therefore, I thought joblib would be an appropriate solution to allow me to use a different directory. However even after specifying: temp_folder='/tmp', I’m getting the following error message:

'NoneType' object has no attribute 'current_process': AttributeError
Traceback (most recent call last):
File "/var/task/world.py", line 74, in handler
r = Parallel(n_jobs=1, verbose=9, temp_folder=temp_folder, backend = 'multiprocessing')(delayed(worker)(i, user_vars) for i in range(user_vars['worlds']))
File "/var/task/deps/joblib/parallel.py", line 728, in __call__
n_jobs = self._initialize_backend()
File "/var/task/deps/joblib/parallel.py", line 540, in _initialize_backend
**self._backend_args)
File "/var/task/deps/joblib/_parallel_backends.py", line 288, in configure
n_jobs = self.effective_n_jobs(n_jobs)
File "/var/task/deps/joblib/_parallel_backends.py", line 268, in effective_n_jobs
if mp.current_process().daemon:
AttributeError: 'NoneType' object has no attribute 'current_process'

mp imports from _multiprocessing_helpers and I must fail at line 27. I dont think I can create an mp.Semaphore()

[Errno 38] Function not implemented: OSError
Traceback (most recent call last):
File "/var/task/world.py", line 74, in handler
mp.Semaphore()
File "/usr/lib64/python2.7/multiprocessing/__init__.py", line 197, in Semaphore
return Semaphore(value)
File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 111, in __init__
SemLock.__init__(self, SEMAPHORE, value, SEM_VALUE_MAX)
File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 75, in __init__
sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 38] Function not implemented

Using backend='threading' works just find, so this is a multiprocessing issue. How can I get around this? Should I be using an environmental variable to define the directory?

[manojkumar1412]

If anyone is still facing this issue, updating sklearn will resolve this. installing scikit-learn using conda version 4.3.14, put a line at 268 of checking mp is none then return 1. which resolve this issue.

[lesteve]

Closing this one, there is nothing joblib can do about this.