inconsistent treatment of None and np.NaN in SimpleImputer

Doing constant imputation treats only the “missing_value” as missing, so a None by default stays there:

from sklearn.impute import SimpleImputer
import numpy as np
X = np.array([1, 2, np.NaN, None]).reshape(-1, 1)
SimpleImputer(strategy='constant', fill_value="asdf").fit_transform()

array([[1],
       [2],
       ['asdf'],
       [None]], dtype=object)

However, using strategy=‘mean’ coerces the None to NaN and so both are replaced:

SimpleImputer(strategy='mean').fit_transform(X)

array([[1. ],
       [2. ],
       [1.5],
       [1.5]])

I don’t think the definition of what’s missing should depend on the strategy. @thomasjpfan argues that the current constant behavior is inconvenient because it means you have to impute both values separately if you want to one-hot-encode.

It seems more safe to treat them differently but I’m not sure there’s a use-case for that. This came up in #17317. I think this only matters in these two, as other imputers don’t allow dtype object arrays.

[jeremiedbb]

mean and median only support numeric arrays, so it’s converted in data validation and None becomes np.nan. In this case passing a list of missing values won’t work. Maybe we could issue a warning when the array has to be converted ?

[thomasjpfan]

On a similar note, setting missing_values=None would raise:

from sklearn.impute import SimpleImputer
import numpy as np

X = np.array([1, 2, np.NaN, None]).reshape(-1, 1)
imputer = SimpleImputer(strategy='constant', fill_value="asdf", 
					    missing_values=None)

# raises
imputer.fit_transform(X)

In the pandas case, since we coerce pd.NA to np.nan the following works:

from sklearn.utils.validation import check_array
import pandas as pd

X_pd =  pd.DataFrame({'f1': pd.Series(['dog', 'cat', pd.NA, None], dtype='category')})

imputer = SimpleImputer(strategy='constant',
					    fill_value='adsf')
imputer.fit_transform(X_pd)

# array([['dog'],
#       ['cat'],
#       ['adsf'],
#       ['adsf']], dtype=object)

This behavior is currently documented in SimpleImputer.

Given we already do coercing to this extend, I would be okay with coercing None to np.nan for object dtypes.