Error if using dask_ml.preprocessing RobustScaler
See original GitHub issueI want to use RobustScaler and got following error:
Traceback (most recent call last):
File "<input>", line 14, in <module>
File "/home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/sklearn/base.py", line 556, in fit_transform
return self.fit(X, y, **fit_params).transform(X)
File "/home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/dask_ml/preprocessing/data.py", line 180, in fit
quantiles = [da.percentile(col, [q_min, 50.0, q_max]) for col in X.T]
File "/home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/dask_ml/preprocessing/data.py", line 180, in <listcomp>
quantiles = [da.percentile(col, [q_min, 50.0, q_max]) for col in X.T]
File "/home/magehex/anaconda3/envs/reinbo/lib/python3.7/site-packages/dask/array/percentile.py", line 133, in percentile
for i, key in enumerate(a.__dask_keys__())
AttributeError: 'numpy.ndarray' object has no attribute '__dask_keys__'
Code:
import numpy as np
from dask_ml.preprocessing import RobustScaler
import time
from sklearn.datasets.samples_generator import make_blobs
# generate 2d classification dataset
X, y = make_blobs(n_samples=100000, centers=2, n_features=380)
X_train = X.astype(np.float32)
y_train = y.astype(np.int32)
start = time.time()
RobustScaler().fit_transform(X_train,y_train)
end = time.time()
print(end - start)
Issue Analytics
- State:
- Created 4 years ago
- Comments:6 (5 by maintainers)
Top Results From Across the Web
dask_ml.preprocessing.RobustScaler - Dask-ML
Scale features using statistics that are robust to outliers. This Scaler removes the median and scales the data according to the quantile range...
Read more >Preprocessing — dask-ml 2022.5.28 documentation
preprocessing contains some scikit-learn style transformers that can be used in Pipelines to perform various data transformations as part of the model fitting ......
Read more >dask_ml.metrics.mean_absolute_percentage_error - Dask-ML
Returns a full set of errors in case of multioutput input. 'uniform_average' : Errors of all outputs are averaged with uniform weight. computebool....
Read more >dask_ml.metrics.mean_squared_error - Dask-ML
Errors of all outputs are averaged with uniform weight. squaredbool, default=True. If True returns MSE value, if False returns RMSE value. Returns.
Read more >API Reference — dask-ml 2022.5.28 documentation
Transforms features using quantile information. preprocessing.Categorizer ([categories, columns]). Transform columns of a DataFrame to categorical dtype.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
I think both are surprising. The results don’t match Scikit-learn’s results in either case.
I like the approach in https://github.com/dask/dask-ml/pull/580#issuecomment-549077034: raise an error saying “use Scikit-learn. If you have to, you can wrap in a Dask array, but that’ll yield slower/approximate results”: https://github.com/dask/dask-ml/blob/7793962d990a4b5290a5cfd1aee1d5b48828f1a1/dask_ml/decomposition/pca.py#L18-L24
That presents some challenges for pipelines with Dask-ML estimators and Dask array inputs. What if
RobustScaler
is an element in that pipeline? Maybe there should be a wrapper class to take Dask inputs, call the corresponding Scikit-learn estimator with a NumPy array and return a Dask output?So, as a user, what’s less surprising?