Investigate mypy errors

Currently running mypy on the code base of the master branch (without any additional type annotations) with,

mypy --ignore-missing-imports sklearn

yields the following errors,

sklearn/externals/joblib/_compat.py:10: error: Name 'basestring' is not defined
sklearn/externals/joblib/_compat.py:11: error: Name 'unicode' is not defined
sklearn/externals/joblib/_compat.py:14: error: Incompatible types in assignment (expression has type "Tuple[Type[bytes], Type[str]]", variable has type "Tuple[Type[str], Any]")
sklearn/externals/six.py:383: error: Cannot determine type of 'next'
sklearn/externals/six.py:391: error: Cannot determine type of 'callable'
sklearn/externals/joblib/my_exceptions.py:69: error: Need type annotation for '_exception_mapping'
sklearn/externals/joblib/disk.py:20: error: Cannot determine type of 'WindowsError'
sklearn/externals/joblib/compressor.py:13: error: Name 'RLock' already defined on line 106
sklearn/externals/joblib/compressor.py:18: error: Incompatible types in assignment (expression has type "None", variable has type Module)
sklearn/externals/joblib/compressor.py:23: error: Incompatible types in assignment (expression has type "None", variable has type Module)
sklearn/externals/joblib/compressor.py:36: error: Need type annotation for '_COMPRESSORS'
sklearn/externals/_arff.py:297: error: Incompatible types in assignment (expression has type "str", base class "ArffException" defined the type as "None")
sklearn/externals/_arff.py:301: error: Incompatible types in assignment (expression has type "str", base class "ArffException" defined the type as "None")
sklearn/externals/_arff.py:315: error: Incompatible types in assignment (expression has type "str", base class "ArffException" defined the type as "None")
sklearn/externals/_arff.py:352: error: Incompatible types in assignment (expression has type "str", base class "ArffException" defined the type as "None")
sklearn/externals/_arff.py:356: error: Incompatible types in assignment (expression has type "str", base class "ArffException" defined the type as "None")
sklearn/externals/_arff.py:360: error: Incompatible types in assignment (expression has type "str", base class "ArffException" defined the type as "None")
sklearn/externals/_arff.py:374: error: Name 'BadObject' already defined on line 367
sklearn/externals/joblib/numpy_pickle_utils.py:18: error: Module has no attribute "_Unpickler"
sklearn/externals/joblib/numpy_pickle_utils.py:19: error: Module has no attribute "_Pickler"
sklearn/externals/joblib/numpy_pickle_utils.py:37: error: Incompatible types in assignment (expression has type "None", variable has type Module)
sklearn/externals/joblib/hashing.py:22: error: Module has no attribute "_Pickler"
sklearn/externals/joblib/hashing.py:52: error: Invalid base class
sklearn/externals/joblib/format_stack.py:35: error: Incompatible types in assignment (expression has type "Callable[[Callable[[], bytes]], Generator[TokenInfo, None, None]]", variable has type "Callable[[Callable[[], str]], Generator[TokenInfo, None, None]]")
sklearn/externals/joblib/backports.py:42: error: All conditional function variants must have identical signatures
sklearn/externals/joblib/_memory_helpers.py:96: error: All conditional function variants must have identical signatures
sklearn/externals/joblib/externals/loky/backend/semlock.py:43: error: Argument 1 to "CDLL" has incompatible type "Optional[str]"; expected "str"
sklearn/externals/joblib/externals/loky/backend/semlock.py:58: error: All conditional function variants must have identical signatures
sklearn/externals/joblib/externals/loky/backend/semlock.py:149: error: Module has no attribute "_RandomNameSequence"
sklearn/externals/joblib/externals/loky/backend/_posix_wait.py:19: error: Incompatible types in assignment (expression has type "None", variable has type Module)
sklearn/externals/joblib/externals/loky/backend/_posix_wait.py:23: error: Argument 1 to "CDLL" has incompatible type "Optional[str]"; expected "str"
sklearn/externals/joblib/externals/loky/backend/_posix_wait.py:24: error: Argument 1 to "CDLL" has incompatible type "Optional[str]"; expected "str"
sklearn/externals/joblib/func_inspect.py:23: error: List or tuple literal expected as the second argument to namedtuple()
sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:75: error: Module has no attribute "ClassType"
sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:76: error: Module 'pickle' has no attribute '_Pickler'
sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:76: error: Name 'Pickler' already defined on line 37
sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:83: error: Need type annotation for '_dynamic_modules_globals'
sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:122: error: Name 'cell' is not defined
sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:270: error: "Type[Pickler]" has no attribute "dispatch"
sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:584: error: Need type annotation for '_extract_code_globals_cache'
sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:695: error: Module has no attribute "ClassType"
sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:854: error: Name 'file' is not defined
sklearn/externals/joblib/numpy_pickle.py:14: error: Cannot assign to a type
sklearn/externals/joblib/numpy_pickle.py:14: error: Incompatible types in assignment (expression has type "None", variable has type "Type[Path]")
sklearn/externals/joblib/_multiprocessing_helpers.py:17: error: Name 'mp' already defined on line 14
sklearn/externals/joblib/_multiprocessing_helpers.py:32: error: "int" has no attribute "get_context"
sklearn/externals/joblib/_multiprocessing_helpers.py:47: error: Name 'assert_spawning' already defined on line 179
sklearn/externals/joblib/_multiprocessing_helpers.py:49: error: Incompatible types in assignment (expression has type "None", variable has type "Callable[[Any], None]")
sklearn/externals/joblib/externals/loky/backend/compat.py:17: error: Module 'multiprocessing.process' has no attribute 'BaseProcess'
sklearn/externals/joblib/_memmapping_reducer.py:20: error: Cannot determine type of 'WindowsError'
sklearn/externals/joblib/_memmapping_reducer.py:24: error: Module 'pickle' has no attribute 'whichmodule'
sklearn/externals/joblib/_memmapping_reducer.py:407: error: The type alias is invalid in runtime context
sklearn/externals/joblib/externals/loky/cloudpickle_wrapper.py:11: error: Need type annotation for 'WRAP_CACHE'
sklearn/externals/joblib/externals/loky/backend/context.py:159: error: Incompatible types in assignment (expression has type "Type[LokyProcess]", base class "BaseContext" defined the type as "Callable[[BaseContext, Any, Optional[Callable[..., Any]], Optional[str], Iterable[Any], Mapping[Any, Any], DefaultNamedArg(Optional[bool], 'daemon')], Process]")
sklearn/externals/joblib/externals/loky/backend/context.py:160: error: Incompatible types in assignment (expression has type "staticmethod", base class "BaseContext" defined the type as "Callable[[BaseContext], int]")
sklearn/externals/joblib/externals/loky/backend/context.py:258: error: Incompatible types in assignment (expression has type "Type[LokyInitMainProcess]", base class "BaseContext" defined the type as "Callable[[BaseContext, Any, Optional[Callable[..., Any]], Optional[str], Iterable[Any], Mapping[Any, Any], DefaultNamedArg(Optional[bool], 'daemon')], Process]")
sklearn/externals/joblib/externals/loky/backend/context.py:264: error: Module has no attribute "_concrete_contexts"
sklearn/externals/joblib/externals/loky/backend/context.py:265: error: Module has no attribute "_concrete_contexts"
sklearn/externals/joblib/externals/loky/backend/spawn.py:13: error: Module 'multiprocessing' has no attribute 'util'
sklearn/externals/joblib/externals/loky/backend/reduction.py:15: error: Module 'multiprocessing' has no attribute 'util'
sklearn/externals/joblib/externals/loky/backend/reduction.py:55: error: Need type annotation for 'dispatch_table'
sklearn/externals/joblib/externals/loky/backend/__init__.py:14: error: "Type[SemLock]" has no attribute "_make_name"
sklearn/externals/joblib/externals/loky/backend/semaphore_tracker.py:32: error: Module 'multiprocessing' has no attribute 'util'
sklearn/externals/joblib/externals/loky/backend/queues.py:18: error: Module 'multiprocessing' has no attribute 'util'
sklearn/externals/joblib/externals/loky/backend/queues.py:20: error: Module 'multiprocessing.synchronize' has no attribute 'SEM_VALUE_MAX'
sklearn/externals/joblib/externals/loky/backend/queues.py:21: error: Module 'multiprocessing.queues' has no attribute 'Full'
sklearn/externals/joblib/externals/loky/backend/queues.py:22: error: Module 'multiprocessing.queues' has no attribute '_sentinel'
sklearn/externals/joblib/externals/loky/backend/popen_loky_win32.py:4: error: Module 'multiprocessing' has no attribute 'util'
sklearn/externals/joblib/externals/loky/backend/synchronize.py:23: error: Module 'multiprocessing' has no attribute 'util'
sklearn/externals/joblib/externals/loky/backend/synchronize.py:61: error: Module has no attribute "_RandomNameSequence"
sklearn/externals/joblib/externals/loky/backend/popen_loky_posix.py:14: error: Module 'multiprocessing' has no attribute 'util'
sklearn/externals/joblib/externals/loky/backend/popen_loky_posix.py:190: error: Need type annotation for 'info'
sklearn/externals/joblib/externals/loky/backend/popen_loky_posix.py:196: error: "Process" has no attribute "_inheriting"
sklearn/externals/joblib/externals/loky/backend/popen_loky_posix.py:202: error: "Process" has no attribute "_inheriting"
sklearn/externals/joblib/pool.py:21: error: Cannot determine type of 'WindowsError'
sklearn/externals/joblib/pool.py:322: error: Parameterized generics cannot be used with class or instance checks
sklearn/externals/joblib/pool.py:322: error: The type alias is invalid in runtime context
sklearn/externals/joblib/externals/loky/backend/managers.py:13: error: Module 'multiprocessing.managers' has no attribute 'State'
sklearn/externals/joblib/externals/loky/process_executor.py:89: error: Cannot assign to a type
sklearn/externals/joblib/externals/loky/process_executor.py:89: error: Cannot assign multiple types to name "_BPPException" without an explicit "Type[...]" annotation
sklearn/externals/joblib/externals/loky/process_executor.py:89: error: Incompatible types in assignment (expression has type "Type[RuntimeError]", variable has type "Type[BrokenProcessPool]")
sklearn/externals/joblib/externals/loky/process_executor.py:111: error: Need type annotation for '_threads_wakeups'
sklearn/externals/joblib/_parallel_backends.py:30: error: Invalid base class
sklearn/externals/joblib/parallel.py:203: error: Item "int" of "Optional[int]" has no attribute "get_context"
sklearn/externals/joblib/parallel.py:203: error: Item "None" of "Optional[int]" has no attribute "get_context"
sklearn/externals/joblib/_dask.py:107: error: Incompatible types in assignment (expression has type "float", base class "AutoBatchingMixin" defined the type as "int")
sklearn/externals/joblib/_store_backends.py:34: error: Invalid base class
sklearn/externals/joblib/memory.py:174: error: Need type annotation for '_FUNCTION_HASHES'
sklearn/utils/fixes.py:141: error: All conditional function variants must have identical signatures
sklearn/utils/fixes.py:177: error: Name 'MaskedArray' already defined on line 163
sklearn/__init__.py:54: error: Cannot determine type of '__SKLEARN_SETUP__'
sklearn/utils/testing.py:36: error: Cannot determine type of 'WindowsError'
sklearn/utils/testing.py:92: error: Module has no attribute "case"
sklearn/utils/_pprint.py:324: error: "Type[PrettyPrinter]" has no attribute "_dispatch"
sklearn/model_selection/_split.py:2232: error: "Callable[[VarArg(Any), KwArg(Any)], Any]" has no attribute "__test__"
sklearn/preprocessing/_encoders.py:289: error: Decorated property not supported
sklearn/preprocessing/_encoders.py:296: error: Decorated property not supported
sklearn/preprocessing/_encoders.py:303: error: Decorated property not supported
sklearn/svm/base.py:8: error: Module 'sklearn.svm' has no attribute 'libsvm'
sklearn/svm/base.py:8: error: Module 'sklearn.svm' has no attribute 'liblinear'
sklearn/svm/base.py:9: error: Module 'sklearn.svm' has no attribute 'libsvm_sparse'
sklearn/svm/__init__.py:16: error: Module 'sklearn.svm' has no attribute 'libsvm'
sklearn/svm/__init__.py:16: error: Module 'sklearn.svm' has no attribute 'liblinear'
sklearn/svm/__init__.py:16: error: Module 'sklearn.svm' has no attribute 'libsvm_sparse'
sklearn/linear_model/stochastic_gradient.py:508: error: Decorated property not supported
sklearn/linear_model/least_angle.py:23: error: Module 'sklearn.utils' has no attribute 'arrayfuncs'
sklearn/linear_model/coordinate_descent.py:28: error: Module 'sklearn.linear_model' has no attribute 'cd_fast'
sklearn/manifold/t_sne.py:27: error: Module 'sklearn.manifold' has no attribute '_barnes_hut_tsne'
sklearn/cluster/k_means_.py:36: error: Module 'sklearn.cluster' has no attribute '_k_means'
sklearn/tests/test_docstring_parameters.py:24: error: Module has no attribute "__path__"
sklearn/manifold/tests/test_t_sne.py:30: error: Module 'sklearn.manifold' has no attribute '_barnes_hut_tsne'
sklearn/tests/test_random_projection.py:30: error: Unsupported operand types for + ("List[Callable[[Any, Any, Any, Any], Any]]" and "List[Callable[[Any, Any, Any], Any]]")
sklearn/tests/test_random_projection.py:34: error: Unsupported operand types for + ("List[Type[SparseRandomProjection]]" and "List[Type[GaussianRandomProjection]]")
sklearn/ensemble/gradient_boosting.py:1354: error: Decorated property not supported
sklearn/covariance/graph_lasso_.py:24: error: Module 'sklearn.linear_model' has no attribute 'cd_fast'
sklearn/model_selection/tests/test_validation.py:84: error: Cannot determine type of 'WindowsError'
sklearn/compose/tests/test_column_transformer.py:815: error: No overload variant of "slice" matches argument types "None", "str"
sklearn/compose/tests/test_column_transformer.py:815: note: Possible overload variant:
sklearn/compose/tests/test_column_transformer.py:815: note:     def __init__(self, start: Optional[int], stop: Optional[int], step: Optional[int] = ...) -> slice
sklearn/compose/tests/test_column_transformer.py:815: note:     <1 more non-matching overload not shown>
sklearn/compose/tests/test_column_transformer.py:815: error: No overload variant of "slice" matches argument types "str", "str"
sklearn/compose/tests/test_column_transformer.py:815: note:     def __init__(self, start: Optional[int], stop: Optional[int], step: Optional[int] = ...) -> slice
sklearn/ensemble/tests/test_forest.py:108: error: Argument 1 to "update" of "dict" has incompatible type "Dict[str, Type[ForestRegressor]]"; expected "Mapping[str, Type[ForestClassifier]]"
sklearn/ensemble/tests/test_forest.py:109: error: Argument 1 to "update" of "dict" has incompatible type "Dict[str, Type[RandomTreesEmbedding]]"; expected "Mapping[str, Type[ForestClassifier]]"
sklearn/ensemble/tests/test_forest.py:112: error: Argument 1 to "update" of "dict" has incompatible type "Dict[str, Type[ForestRegressor]]"; expected "Mapping[str, Type[ForestClassifier]]"
sklearn/ensemble/tests/test_forest.py:1311: error: Invalid base class

while most of these can probably be ignored (or are due to vendored packages), some might be worth investigating.

Related to https://github.com/scikit-learn/scikit-learn/issues/11170

[rth]

Thanks @nixphix !

I hope its safe to ignore errors reported under externals/* or should I investigate them and report to respective vendor repo ?

This issue is only about warnings/errors from scikit-learn, but if you see things worth reporting in vendored projects it’s up to you if you want to report them upstream…

[deeplook]

I’d recommend using mypy more even without type annotations, as I’m doing here: https://github.com/scikit-learn/scikit-learn/pull/15220.