TfidfVectorizer handles multiple text columns

It’s really nice that transformers such as sklearn.preprocessing.OneHotEncoder and sklearn.preprocessing.StandardScaler can operate on multiple data columns simultaneously.

sklearn.feature_extraction.text.TfidfVectorizer on the other hand, can only process one column at a time, so you need to make a new transformer for each text column in your dataset. This can get a little tedious and in particular makes pipelines more verbose.

It’d be nice if TfidfVectorizer could also operate on multiple text columns, using the same settings for each column, perhaps with an option to make one vocabulary per column, or use a shared vocabulary across all the columns.

It might be easiest to implement this as a new class that wraps TfidfVectorizer sagemaker-scikit-learn-extension takes this approach.

If this seems like a good idea, I’d be happy to make a PR.

[zachmayer]

@jnothman @amueller Here’s an example pipeline I was debugging today. It follows a common failure pattern I’ve seen a few times now:

1. Categorical preprocessing handles multiple columns
2. Numeric preprocessing handles multiple columns
3. Text preprocessing handles multiple columns oops

# Categorical pipeline
categorical_preprocessing = Pipeline(
[
    ('Imputation', SimpleImputer(strategy='constant', fill_value='?')),
    ('One Hot Encoding', OneHotEncoder(handle_unknown='ignore')),
]
)
# Numeric pipeline
numeric_preprocessing = Pipeline(
[
     ('Imputation', SimpleImputer(strategy='mean')),
     ('Scaling', StandardScaler())
]
)
text_preprocessing = Pipeline(
[
     ('Text',TfidfVectorizer())       
]
)
# Creating preprocessing pipeline
preprocessing = make_column_transformer(
     (numeric_preprocessing, numeric_features),
     (categorical_preprocessing, categorical_features),
     (text_preprocessing, text_features),
)
# Final pipeline
pipeline = Pipeline(
[('Preprocessing', preprocessing)]
)
test = pipeline.fit_transform(x_train)

I fixed this bug by replacing TfidfVectorizer with sagemaker_sklearn_extension.feature_extraction.text import MultiColumnTfidfVectorizer.

The fact that AWS added this to their sagemaker_sklearn_extension extensions indicate their users frequently run into this problem too.

[zachmayer]

I’ve used SageMaker’s TFIDFVectorizer and I like it. It’s nice and simple. Doesn’t support all the parameters of TFIDFVectorizer in sklearn though which is annoying