Lime with pipeline

May you show a case how to use LIME with Pipeline? I do

explainer = lime.lime_tabular.LimeTabularExplainer(union, feature_names=names, 
class_names=y_train, categorical_features=cat, verbose=True, mode='regression')

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().

Where union is sparse matrix obtained from FeatureUnion. Pipeline as below

pipeline = Pipeline([

    # Use FeatureUnion to combine the features
    ('union', FeatureUnion(
        transformer_list=[ 
          # text
            ('text', Pipeline([
                ('select',  norm()),
                ('tfidf', TfidfVectorizer(max_df=40, min_df=3, ngram_range=(1, 4)))
            ])),  
                     # categorical
            ('categorical', Pipeline([
                                 ('selector', MultiColumn(columns=['a', 'b'])),
                ('one_hot', Cat())
            ])),
            # numeric
            ('numeric', Pipeline([
                 ('date', Age()),
                 ('scaling', preprocessing.MinMaxScaler())
            ])),
        ])),
    # Use a regression
    ('model_fitting',  xgb),
])

I extract feature names as below one = pipeline.named_steps['union'].transformer_list[0][1].named_steps['tfidf'].get_feature_names() for text. Then names of categorical variables

enc = DictVectorizer(sparse = False)
cat= X[['a', 'b,]]
enccat = enc.fit((cat.reset_index(drop=True)).T.to_dict().values())
two = enccat.get_feature_names()

and for numeric just

three = ['age']

Then i get array of feature names

names = one+two+three
names = np.asarray(ferche_names)
cat = np.asarray(two)

I get sparse matrix just using part FeatureUnion from Pipeline union = union.fit_transform(X_train)

[marcotcr]

LimeTabularExplainer assumes the input matrix is dense, and that each column represents a feature (i.e. it’s not one-hot encoded). Unfortunately this would require you to break the pipeline in two, and letting the explainer do the one-hot encoding. (reopen if this doesn’t answer your question)

[biomed-dmnlp]

LimeTabularExplainer assumes the input matrix is dense, and that each column represents a feature (i.e. it’s not one-hot encoded). Unfortunately this would require you to break the pipeline in two, and letting the explainer do the one-hot encoding. (reopen if this doesn’t answer your question)

Hi Marcotcr, may you explain how to break the pipeline into two? I am also interested in how to explain the model using mixed types of data (categorical and text). thanks.