Design of add_indicator in SimpleImputer may fail when running cross validation

Description

The design of add_indicator depends on missing values exist in the training data. This will break cross validation.

Steps/Code to Reproduce

import numpy as np
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import PredefinedSplit
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

X = np.array([[1, 2, 3, np.nan]]).T
y = np.array([0, 0, 1, 1])
test_fold = np.array([0, 1, 0, 1])

ps = PredefinedSplit(test_fold)
pipe1 = make_pipeline(SimpleImputer(add_indicator=True), 
                      LogisticRegression(solver='lbfgs'))

cross_val_score(pipe1, X, y, cv=ps)

Expected Results

No error is thrown.

Actual Results

ValueError: The features [0] have missing values in transform 
but have no missing values in fit.

Thoughts

The original design was adopted because, if the training data has no missing value, there will be a column with all zeros. This type of error will appear when we try to do grid search over the add_indicator parameter. One way to work around this is to split the data in such a way that missing values are available (for each column that has a missing value) in both the training set and test set.

[amueller]

@pmattioli for the MissingIndicator that’s constructed in SimpleImputer.

[pmattioli]

We should have error_on_new=False by default for add_indicator.

@jnothman do you mean we should have error_on_new=False for MissingIndicator?