Multiprocessing backed parallel loops cannot be nested below threads, setting n_jobs=1

What is the reason of such issue in joblib? 'Multiprocessing backed parallel loops cannot be nested below threads, setting n_jobs=1' What should I do to avoid such issue?

Actually I need to implement XMLRPC server which run heavy computation in background thread and report current progress through polling from UI client. It uses scikit-learn which are based on joblib.

P.S.: I’ve simply changed name of the thread to “MainThread” to avoid such warning and everything looks working good (run in parallel as expected without issues). What might be a problem in future for such workaround?

[rth]

While using multiprocessing below threads is not supported, the current test to detect it relies on the thread name being equal to MainThread which can produce false positives.

For instance, I’m trying to use joblib parallel inside uWSGI (in a non multi-threaded setup). Different workers use a single thread, but the default thread name is something of the form uWSGIWorker2Core0, and renaming the thread name to MainThread appears to make everything work fine (same as for the OP), though I hope this doesn’t break anything in uWSGI.

As suggested in this SO answer, a more robust testing for multiprocessing nested under threads in joblib could be,

isinstance(threading.current_thread(), threading._MainThread))

instead of the currently used,

threading.current_thread().name == 'MainThread'

For my uWSGI setup, the latter evaluates to False, while the former is True.

[rth]

This was fixed a long time ago and renaming the thread to MainThread should no longer be necessary. Please open a new issue of you still encounter difficulties with this.