TypeError: ufunc 'isnan' not supported for the input types

Code to reproduce the error

df = pd.read_csv('/home/shahul/Downloads/train.csv.zip').sample(10000)
y = df['target']
X = df.drop(['target'],axis=1)

a = AutoML(total_time=30,tuning_mode="Normal")

a.fit(X, y)

the error happens due to the use of np.isna() to object dtype which happens in np.nanmedian() used in

/mljar-supervised/supervised/preprocessing/preprocessing_utils.py

here I have used PNB Paribas dataset as train

[pplonski]

@abtheo thank you for explanations! I see the problem right now.

[abtheo]

Practically, to handle objects of unknown content, I see three main use cases we need to cover:

#To be encoded as CATEGORICAL
1. strs_as_object = np.array(["A", "B", "C"], dtype=object)

#To be encoded as DISCRETE (or equivalently, CONTINUOUS for floats)
2. nums_as_object = np.array([1, 2, "3"], dtype=object)

#To be encoded as CATEGORICAL
3. mixed_input_object = np.array([1, "B", 3], dtype=object)

Currently, Cases 1&2 work as expected, however we are not handling Case 3. The ambiguous object type causes Numpy issues all over the place.

As one example, using Case 3 as the input to AutoML.fit(y=mixed_input_object) causes the following error to occur at Line 44 of preprocessing_utils.py :

unique_cnt = len(np.unique(x[~pd.isnull(x)]))

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
TypeError: '<' not supported between instances of 'str' and 'int'

As another example, using Case 3 as the input to AutoML.fit(x=mixed_input_object) causes problems with to_parquet(), as seen here: https://github.com/pandas-dev/pandas/issues/21228

To solve both of these issues, the very first thing we should do is validate the type of the data. Here is my proposed solution:

y_train_type = PreprocessingUtils.get_type(y_train)

if y_train_type == PreprocessingUtils.DISCRETE or y_train_type == PreprocessingUtils.CONTINUOUS:
            y_train = pd.to_numeric(y_train, errors='coerce')

if y_train_type == PreprocessingUtils.CATEGORICAL:
            y_train = pd.Series([str(y) for y in y_train], name="target")