GridSearchCV fails with XGBoost models

The following code fails:

import numpy as np
from dask_ml.model_selection import GridSearchCV, KFold
from dask_xgboost import XGBClassifier

x = np.random.randn(100, 2)
y = np.random.randint(0, 1, 100)

params = {'max_depth': [2, 3]}

clf = GridSearchCV(XGBClassifier(), params, cv=KFold(n_splits=2), scoring='neg_mean_squared_error')
clf.fit(x, y)

Stack trace:

  File "/home/jin/anaconda3/envs/ml-gpu/lib/python3.6/site-packages/dask_ml/model_selection/_normalize.py", line 38, in normalize_estimator
    val = getattr(est, attr)
  File "/home/jin/anaconda3/envs/ml-gpu/lib/python3.6/site-packages/xgboost/sklearn.py", line 536, in feature_importances_
    b = self.get_booster()
  File "/home/jin/anaconda3/envs/ml-gpu/lib/python3.6/site-packages/xgboost/sklearn.py", line 200, in get_booster
    raise XGBoostError('need to call fit or load_model beforehand')
xgboost.core.XGBoostError: need to call fit or load_model beforehand

The error arises because the normalize_estimator function is trying to get the ‘feature_importances_’ attribute of the xgb model, but the attribute doesn’t exist unless the model has already been trained. The normalize_estimator function I believe should gloss over this error, but it doesn’t because it’s not catching the XGBoostError error type.

[rrpelgrim]

I’m running into the same issue today. Do I understand correctly that there’s no way to implement dask_xgboost.XGBoostClassifier with dask_ml.model_selection.GridSearchCV ?

[TomAugspurger]

dask_xgboost.XGBoostClassifier doesn’t implement out-of-core training. It loads the data into distributed memory, and hands that off to xgboost’s distributed runtime.

My understanding is that the native xgboost.dask.XGBoostClassifier does the same thing.

On Mon, Jun 8, 2020 at 9:55 PM Gideon Blinick [e-mail] wrote:

Hi @trivialfis https://github.com/trivialfis,

The reason we’d want to use dask’s XGBClassifier is to avoid loading all data into memory at once.

Dask XGBoost seems to have a memory leakage issue though.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/dask-ml/issues/667#issuecomment-640997713, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKAOITK6VKNEEPW5D2YYRLRVWQD3ANCNFSM4NBYBPNA .