dict_learning and partial_dependence functions can be shadowed by deprecated module imports

Since scikit-learn 0.22, the dict_learning function is defined in sklearn/decomposition/_dict_learning.py and exposed as sklearn.decomposition.dict_learning in the sklearn/decomposition/__init__.py file.

At the same time we also have a sklearn/decomposition/dict_learning.py deprecated module for backward compat. This module is not imported by default in sklearn/decomposition/__init__.py to avoid deprecation warning.

So far so good. The problem is that if the users or a tool later import sklearn.decomposition.dict_learning (the deprecated module), then the symbol sklearn.decomposition.dict_learning points to the module instead of the function:

>>> from sklearn.decomposition import dict_learning                                                                                                                             
>>> type(dict_learning)                                                                                                                                                         
<class 'function'>
>>> import sklearn.decomposition.dict_learning                                                                                                                                  
/home/ogrisel/code/scikit-learn/sklearn/utils/deprecation.py:144: FutureWarning: The sklearn.decomposition.dict_learning module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.decomposition. Anything that cannot be imported from sklearn.decomposition is now part of the private API.
  warnings.warn(message, FutureWarning)
>>> from sklearn.decomposition import dict_learning                                                                                                                             
>>> type(dict_learning)                                                                                                                                                         
<class 'module'>

This can happen (semi-randomly?) when running test discovery with pytest --pyargs sklearn from a non-source folder (without the scikit-learn conftest.py file) or when calling all_estimators() from scikit-learn.

Note that in 0.21.3 we did not have the problem because once the sklearn.decomposition.dict_learning module is imported once, it is cached, and therefore, sklearn.decomposition.dict_learning is always referring to the function.

We have the same problem for the sklearn.inspection.partial_dependence function / module.

I will open a PR with a possible fix.

[glemaitre]

If I recall correctly, it depends on the pytest version as well: https://github.com/scikit-learn/scikit-learn/blob/master/azure-pipelines.yml#L60

[ogrisel]

I realized that I did not answer this question: in 0.21. Let me answer it now for future reference:

In 0.21, doing from sklearn.decomposition import dict_learning you actually trigger the import of the sklearn.decomposition.dict_learning module before setting the attribute sklearn.decomposition.dict_learning to the function.

If after you do from sklearn.decomposition.dict_learning import dict_learning, the sklearn.decomposition.dict_learning is already imported and in the sys.modules cache, therefore nothings happen and the sklearn.decomposition.dict_learning attribute still points to the function.