Attention layer output

The method task_specific_attention applies attention to the projected vectors instead of the hidden vectors (output from RNN cell).

Has it been applied purposefully or has the information on attention according to the paper been missed out where final sentence vector is weighted summation of hidden states and NOT inner projected vector?

[krayush07]

@heisenbugfix I agree to what you mentioned in the previous comment. However, final attention is applied on hidden states and NOT projected vector.

As per my understanding, here are the steps apply attention:

1. Collect ‘hidden_states’
2. Apply projection to get projected vector.
3. Use projected vector and attention vector to find attention weights.
4. Use attention weights and hidden_states to apply attention.

I find a mismatch in 4th step in your code. Please correct me if I am wrong.

[momih]

@krayush07 I think it’s more of a personal choice where to apply attention weights. In the paper, the authors project the hidden state to the same dimension and then compute attention and apply it to the hidden state. However, in this implementation he projects the hidden state to a lower dimension to compute attention. So I’m guessing he applies attention to the projected vector instead because he wants a lower dimension for the encoded sentence vector.