Why choose model.embeddings over model.embeddings.word_embeddings when calculating LayerIntegratedGradients for Bert like models.

Hi,

I’ve been playing with the idea of only using a models’ embeddings.word_embeddings when calculating LIG for Bert like models rather than embeddings. However I have noticed that with some models the attribution scores can change quite a bit for certain words.

I know that for a model like Bert the embeddings are often broken down into word_embeddings, token_type_embeddings, and position_embeddings. However, the only embedding layer that returns meaningful results attribution results is word_embeddings, if I use the token or position embeddings I get nan for all word attributions.

What I was wondering is if there is any reason for models like bert, distilbert, etc to calculate LIG attributions along all the embedding rather than just word ?

Let me know if you need any clarification on what I’m trying to ask, I hope it’s not too vague. Thanks !

[cdpierse]

@NarineK Ah, this worked perfectly for me, I should have seen that inputs can take multiple values. Thanks for helping me with this, it’s a huge help. Much appreciated ! Do you think the original SQUAD example or notebook would benefit from this addition ? I could put together a PR if you think so.

[cdpierse]

@NarineK thanks for the response. I had see that example before but I think I’ve put together why I was getting such weird responses, I wasn’t testing very thoroughly apparently. I’ve been using the distilbert model as an example model for lot of these tasks, primarily because it’s smaller faster to test on etc. The problem is that distilbert doesn’t accept position_ids or token_ids as inputs for its forward pass unlike most other model configs in the HF library. Thanks again for the response!