LIME explain instance resulting in empty graph

Hi. I’m working with a spark data frame and to be able to make use of LIME we had to make some modifications:

def new_predict_fn(data):
    sdf = map(lambda x: (int(x[0]), Vectors.dense(x[0:])), data)
    sdf = spark.createDataFrame(sdf, schema=["id", "features"]).select("features")
    predictions = cv_model.transform(sdf).select("prediction")
    return predictions.toPandas()["prediction"].values.reshape(-1)

lime_df_test = nsdf.select('features').toPandas()
lime_df_test = pd.DataFrame.from_records(lime_df_test['features'].tolist())

exp = explainer.explain_instance(lime_df_test.iloc[10].values, new_predict_fn, num_features=20)
display(exp.as_pyplot_figure())

However, when it results in an “empty” explanation:

[(‘3575 <= 2199.13’, 0.0), (‘3981 <= 2189.88’, 0.0), (‘3987 <= 2189.88’, 0.0), (‘4527 <= 93.00’, 0.0), (‘4003 <= 1.00’, 0.0), (‘4528 <= 0.00’, 0.0), (‘3824 <= 14000000.00’, 0.0), (‘4256 <= 2199.73’, 0.0), (‘3685 <= 2190.45’, 0.0), (‘3579 <= 2199.13’, 0.0)]

We are looking for some reason for this to happen. A simple test with the modifications mentioned above worked well, but using real data (with more than 3000) columns, we faced that problem. The only idea that comes to my mind is that LIME is not being able to explain an instance locally (?). But I’m not sure if that makes sense. I’m also wondering (now) if it’s not just a case that the weights are plotted with 1 decimal place of precision and if (how) I could change that.

Thanks.

[hanzigs]

Hi, I am having similar issue getting zeros in binary classification, Can I have some help please, Thanks

X_train.shape
Out[180]: (152491, 165)
X_test[180].shape
Out[181]: (165,)
rf_explainer = lime.lime_tabular.LimeTabularExplainer(X_train, mode='classification',training_labels=data['class'],feature_names=feature_names)
exp = rf_explainer.explain_instance(X_test[180], rf_fit.predict_proba, num_features=10)
exp.as_list()
Out[183]: 
[('Contact_Y <= 1.00', 0.0),
 ('Business_Name_Y <= 1.00', 0.0),
 ('Y1SA_Y <= 1.00', 0.0),
 ('Valuation_Acceptable_Valuation_Acceptable_miss <= 1.00', 0.0),
 ('Trading_State_WA <= 1.00', 0.0),
 ('Employer_Name_Y <= 1.00', 0.0),
 ('Trading_Unit_Number_Y <= 0.00', 0.0),
 ('Home_State_WA <= 1.00', 0.0),
 ('Applicant_Type_Applicant_Type_miss <= 1.00', 0.0),
 ('Phone_Number_N <= 0.00', 0.0)]

[marcotcr]

exp.as_list() returns explanations for label 1 (that is the default parameter), i.e. positive is positive towards 1 and negative is negative towards 1. You can use exp.as_list(label=2) or any other value if you care about a particular label. 1 is a good default for binary classification, top_labels=1 is a good default otherwise.