Exception in thread QueueManagerThread

Description

When I set n_jobs=-1 I get error and if I set n_jobs equal big value (n_jobs=100), but if set smaller value (e.g. n_jobs=32), it works fine. I’ve tried reinstall scikit-learn and joblib packages, but to no avail. Also, it (n_jobs=-1) works fine previously, but suddenly go wrong. Maybe, this error isn’t related to scikit-learn directly, but rather to threading, joblib and etc.

Steps/Code to Reproduce

from sklearn import datasets
from sklearn.model_selection import cross_validate, StratifiedKFold
from sklearn.linear_model import RidgeClassifier

iris = datasets.load_iris()
iris_X = iris.data
iris_y = iris.target

skf = StratifiedKFold(n_splits=5, random_state=0, shuffle=True)
scoring = {'accuracy': 'accuracy'}

model_ridge = RidgeClassifier(random_state=0)
scores = cross_validate(estimator=model_ridge, 
                            X=iris_X,
                            y=iris_y,
                            scoring=scoring,
                            cv=skf.split(iris_X, iris_y),
                            n_jobs=-1)

Expected Results

Cv scores of ridge model.

Actual Results

Exception in thread QueueManagerThread:
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\py36\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\ProgramData\Anaconda3\envs\py36\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\sklearn\externals\joblib\externals\loky\process_executor.py", line 615, in _queue_management_worker
    ready = wait(readers + worker_sentinels)
  File "C:\ProgramData\Anaconda3\envs\py36\lib\multiprocessing\connection.py", line 859, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "C:\ProgramData\Anaconda3\envs\py36\lib\multiprocessing\connection.py", line 791, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 66

Versions

import sklearn; sklearn.show_versions()
System:
    python: 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 18:50:55) [MSC v.1915 64 bit (AMD64)]
executable: C:/ProgramData/Anaconda3/envs/py36/python.exe
   machine: Windows-10-10.0.14393-SP0

BLAS:
    macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
  lib_dirs: C:/ProgramData/Anaconda3/envs/py36/Library/lib
cblas_libs: mkl_rt

Python deps:
       pip: 18.1
setuptools: 40.6.3
   sklearn: 0.20.1
     numpy: 1.15.0
     scipy: 1.1.0
    Cython: 0.29.5
    pandas: 0.24.1
    joblib: 0.13.2

[rth]

This is a similar issue to https://github.com/scikit-learn/scikit-learn/issues/12263, see https://github.com/tomMoral/loky/issues/192 for the upstream joblib/loky issue.

[vladserkoff]

According to this python bug where’s an some cap on the maximium number of threads in ProcessPoolExecutor under Windows:

Some quick Googling strongly suggests there’s no reasonably general way to overcome the Windows-defined MAXIMUM_WAIT_OBJECTS=64 for implementations that call the Windows WaitForMultipleObjects().