AttributeError: 'AutoSklearnClassifier' object has no attribute 'load_models'

Describe the bug

AttributeError: ‘AutoSklearnClassifier’ object has no attribute ‘load_models’ when trying to score the model with cross validator.

To Reproduce

Run the following code with some dataset in X and y:

from autosklearn.classification import AutoSklearnClassifier
from autosklearn.metrics import average_precision

if __name__ == '__main__':
    clf = AutoSklearnClassifier(metric=average_precision, memory_limit=192*1024)
    clf.fit(X, y)
    cross_validate(clf, X, y, n_jobs=-1, scoring="average_precision")["test_score"].mean()

Expected behavior

I would expect a number to result from it.

Actual behavior, stacktrace or logfile

Stacktrace:

AttributeError                            Traceback (most recent call last)
<ipython-input-29-8d5e33cc8853> in <module>
----> 1 cross_validate(clf, Xfloat, y, n_jobs=-1, scoring="average_precision")["test_score"].mean()

~/SageMaker/custom-miniconda/miniconda/envs/python_38/lib/python3.8/site-packages/sklearn/utils/validation.py in inner_f(*args, **kwargs)
     61             extra_args = len(args) - len(all_args)
     62             if extra_args <= 0:
---> 63                 return f(*args, **kwargs)
     64 
     65             # extra_args > 0

~/SageMaker/custom-miniconda/miniconda/envs/python_38/lib/python3.8/site-packages/sklearn/model_selection/_validation.py in cross_validate(estimator, X, y, groups, scoring, cv, n_jobs, verbose, fit_params, pre_dispatch, return_train_score, return_estimator, error_score)
    244     parallel = Parallel(n_jobs=n_jobs, verbose=verbose,
    245                         pre_dispatch=pre_dispatch)
--> 246     results = parallel(
    247         delayed(_fit_and_score)(
    248             clone(estimator), X, y, scorers, train, test, verbose, None,

~/SageMaker/custom-miniconda/miniconda/envs/python_38/lib/python3.8/site-packages/joblib/parallel.py in __call__(self, iterable)
   1039             # remaining jobs.
   1040             self._iterating = False
-> 1041             if self.dispatch_one_batch(iterator):
   1042                 self._iterating = self._original_iterator is not None
   1043 

~/SageMaker/custom-miniconda/miniconda/envs/python_38/lib/python3.8/site-packages/joblib/parallel.py in dispatch_one_batch(self, iterator)
    829                 big_batch_size = batch_size * n_jobs
    830 
--> 831                 islice = list(itertools.islice(iterator, big_batch_size))
    832                 if len(islice) == 0:
    833                     return False

~/SageMaker/custom-miniconda/miniconda/envs/python_38/lib/python3.8/site-packages/sklearn/model_selection/_validation.py in <genexpr>(.0)
    246     results = parallel(
    247         delayed(_fit_and_score)(
--> 248             clone(estimator), X, y, scorers, train, test, verbose, None,
    249             fit_params, return_train_score=return_train_score,
    250             return_times=True, return_estimator=return_estimator,

~/SageMaker/custom-miniconda/miniconda/envs/python_38/lib/python3.8/site-packages/sklearn/utils/validation.py in inner_f(*args, **kwargs)
     61             extra_args = len(args) - len(all_args)
     62             if extra_args <= 0:
---> 63                 return f(*args, **kwargs)
     64 
     65             # extra_args > 0

~/SageMaker/custom-miniconda/miniconda/envs/python_38/lib/python3.8/site-packages/sklearn/base.py in clone(estimator, safe)
     72 
     73     klass = estimator.__class__
---> 74     new_object_params = estimator.get_params(deep=False)
     75     for name, param in new_object_params.items():
     76         new_object_params[name] = clone(param, safe=False)

~/SageMaker/custom-miniconda/miniconda/envs/python_38/lib/python3.8/site-packages/sklearn/base.py in get_params(self, deep)
    193         out = dict()
    194         for key in self._get_param_names():
--> 195             value = getattr(self, key)
    196             if deep and hasattr(value, 'get_params'):
    197                 deep_items = value.get_params().items()

AttributeError: 'AutoSklearnClassifier' object has no attribute 'load_models'

Environment and installation:

Please give details about your installation:

• OS: AWS Linux
• Using a Conda environment
• Python: 3.8
• Auto-sklearn version: 0.12.1

[mfeurer]

This was fixed via #1063 and will be present in the next release we’re currently preparing via #1081.

[eafpres]

I have the same error.

System: WSL2 (ubuntu) Code:

import autosklearn.classification
import sklearn.model_selection
import sklearn.datasets
import sklearn.metrics
X, y = sklearn.datasets.load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = \
sklearn.model_selection.train_test_split(X, y, random_state=1)
automl = autosklearn.classification.AutoSklearnClassifier()
automl.fit(X_train, y_train)
y_hat = automl.predict(X_test)
print(“Accuracy score”, sklearn.metrics.accuracy_score(y_test, y_hat))

Error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/eafpres/.local/lib/python3.6/site-packages/sklearn/base.py", line 260, in __repr__
    repr_ = pp.pformat(self)
  File "/usr/lib/python3.6/pprint.py", line 144, in pformat
    self._format(object, sio, 0, 0, {}, 0)
  File "/usr/lib/python3.6/pprint.py", line 161, in _format
    rep = self._repr(object, context, level)
  File "/usr/lib/python3.6/pprint.py", line 393, in _repr
    self._depth, level)
  File "/home/eafpres/.local/lib/python3.6/site-packages/sklearn/utils/_pprint.py", line 181, in format
    changed_only=self._changed_only)
  File "/home/eafpres/.local/lib/python3.6/site-packages/sklearn/utils/_pprint.py", line 425, in _safe_repr
    params = _changed_params(object)
  File "/home/eafpres/.local/lib/python3.6/site-packages/sklearn/utils/_pprint.py", line 91, in _changed_params
    params = estimator.get_params(deep=False)
  File "/home/eafpres/.local/lib/python3.6/site-packages/sklearn/base.py", line 195, in get_params
    value = getattr(self, key)
AttributeError: 'AutoSklearnClassifier' object has no attribute 'load_models'

I decided to look at the attributes, and I see there is a _load_models:

>>> dir(automl)
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_check_n_features', '_get_automl_class', '_get_param_names', '_get_tags', '_load_models', '_metric', '_more_tags', '_n_jobs', '_repr_html_', '_repr_html_inner', '_repr_mimebundle_', '_scoring_functions', '_validate_data', 'automl_', 'build_automl', 'cv_results_', 'dask_client', 'delete_output_folder_after_terminate', 'delete_tmp_folder_after_terminate', 'disable_evaluator_output', 'ensemble_nbest', 'ensemble_size', 'exclude_estimators', 'exclude_preprocessors', 'fANOVA_input_', 'fit', 'fit_ensemble', 'get_configuration_space', 'get_models_with_weights', 'get_params', 'get_smac_object_callback', 'include_estimators', 'include_preprocessors', 'initial_configurations_via_metalearning', 'logging_config', 'max_models_on_disc', 'memory_limit', 'metadata_directory', 'n_jobs', 'output_folder', 'per_run_time_limit', 'predict', 'predict_proba', 'refit', 'resampling_strategy', 'resampling_strategy_arguments', 'score', 'seed', 'set_params', 'show_models', 'smac_scenario_args', 'sprint_statistics', 'time_left_for_this_task', 'tmp_folder', 'trajectory_']