Parallel hangs when /dev/shm is too small

Hey guys, I’ve encountered hangs on Parallel usage by sklearn, and one thing I’ve noticed is that hangs are happening when my dataset is big enough. Debugging print showed me that main process is stuck here, so it looks like main process can never get a response from child process. After some code analysis I was able to find usage of /dev/shm in joblib. Output of df showed me that this volume is full, so I gave it more space and after that hangs stopped.

So my suspicion is that joblib doesn’t work with /dev/shmem fully correct, in sense that maybe we should check if there is enough space available and raise an error instead of hangs.

I ran my experiments in Jupyter Notebook, Ubuntu 14.04 in Docker. All libs are latest versions from pip.

If you need any additional info from me, let me know; I will be glad to help.

P.S. There is a bunch of issues with similar symptom here, but I wasn’t sure if they are really related, so I created another one with concrete description.

[Piatachock]

I’m not sure if this is the case, because by “hang” I meant exactly 0% CPU consumption of all subprocesses for like 3 hours minimum (I turned it off after that). IMHO case with swapping should look diffirently.

[Piatachock]

@ogrisel yes, I never tried to change temp_folder parameter.

I think that fact of /dev/shm was full is no surprise, because I’ve run Jupyter in Docker container, and it has default size of /dev/shm equal to 64M. After I increased it, hangs stopped.

Why this case (full shared memory) is leading to hangs is interesting, though.