Relationship between LayerIntegratedGradients & configure_interpretable_embedding_layer

Hi, I am working with the tutorial here. I took the example

question, text = “What is important to us?”, “It is important to us to include, empower and support humans of all kinds.”

and tried to see the effect of the word “What” on the start position using attributions_start_sum[1] which is

tensor(0.3861, device=‘cuda:0’, dtype=torch.float64, grad_fn=<SelectBackward>) Then I ran the code to get the effect of word, token_type and position embeddings and the attributions look like

for “What”

0.02 + (-0.01) + 0.07 != 0.3861

Is this expected ? Could you please help me understand this

[NarineK]

@LopezGG , I’d also look into LayerNorm and Dropout that is happening after concatenating those layers. https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L213

If you attribute to the inputs of the LayerNorm you might get the result that you’re expecting ? Let me know if you get chance to try that.

[LopezGG]

Not looking into this right now. thank you Narine