Sklearn pipeline and cross_val_score don't work for some transformers

Hi,

from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import GradientBoostingClassifier
from dask_ml.decomposition import PCA
from dask_ml.wrappers import ParallelPostFit
from dask_ml.preprocessing import StandardScaler

clf = ParallelPostFit(estimator=GradientBoostingClassifier(), scoring='accuracy')

pipe = make_pipeline(PCA(),clf)
pipe = make_pipeline(StandardScaler(),clf)

mysc = cross_val_score(pipe, dataset.iloc[:,:-1], dataset.iloc[:,-1])

Work for pipe = make_pipeline(StandardScaler(),clf) but not for pipe = make_pipeline(PCA(),clf) Got the error: AttributeError: ‘DataFrame’ object has no attribute ‘chunks’

If I use RobustScaler() instead of StandardScaler(): AttributeError: ‘int’ object has no attribute ‘ndim’

How can I fix this problem?

[TomAugspurger]

You should pass ndarrays to ParallelPostFit.fit if that’s what the estimator (SVC in this case) is expecting.

On Sat, Nov 2, 2019 at 9:13 AM magehex [e-mail] wrote:

@TomAugspurger https://github.com/TomAugspurger Yes, I know that. This is inconvenient. But it seems there is also an other bug in ParallelPostFit:

from sklearn.ensemble import GradientBoostingClassifier from sklearn.svm import SVC from dask_ml.wrappers import ParallelPostFit from dask_ml.datasets import make_classification

X, y = make_classification(n_samples=1000, random_state=0, chunks=1000)

clf = ParallelPostFit(estimator=SVC(), scoring=‘accuracy’) #clf = ParallelPostFit(estimator=GradientBoostingClassifier(), scoring=‘accuracy’) clf.fit(X,y)

Get error: TypeError: unhashable type: ‘Array’

If I use GradientBoostingClassifier() it works

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/dask-ml/issues/578?email_source=notifications&email_token=AAKAOIXN7STZSGDLMEWWXNTQRV4GPA5CNFSM4JIAHUH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC43TMY#issuecomment-549042611, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKAOIX45ENUZU542BNPQWTQRV4GPANCNFSM4JIAHUHQ .

[magehex]

@TomAugspurger Yes, I know that. This is inconvenient. But it seems there is also an other bug in ParallelPostFit:

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.svm import SVC
from dask_ml.wrappers import ParallelPostFit
from dask_ml.datasets import make_classification

X, y = make_classification(n_samples=1000, random_state=0, chunks=1000)

clf = ParallelPostFit(estimator=SVC(), scoring='accuracy')
#clf = ParallelPostFit(estimator=GradientBoostingClassifier(), scoring='accuracy')
clf.fit(X,y)

Get error: TypeError: unhashable type: ‘Array’

If I use GradientBoostingClassifier() it works