What's the consideration of not apply positional encoding for V at the self-attention layer?

Question: What’s the consideration of not apply positional encoding for V at the self-attention layer?

def forward_post(self, 
                     src, 
                     src_mask: Optional[Tensor] = None,
                     src_key_padding_mask: Optional[Tensor] = None,
                     pos: Optional[Tensor] = None):

        q = k = self.with_pos_embed(src, pos)
        src2 = self.self_attn(q, k, value=src, attn_mask=src_mask,
                              key_padding_mask=src_key_padding_mask)[0]

[szagoruyko]

Hi, we follow the standard practice, see Transformer XL or Stand-alone self-attention in vision, in adding positional encoding to queries and keys only, except in our case it is absolute and not relative.

[alexzeng1206]

+1. I have read the code and find that although the positional encoding is added within q and k during the computation of self/cross-attention, the features are just obtained from v (only contain appearance feature without positional encoding) throughout the process, so I don’t know why the final output slot contains spatial information that allows the FFN to predict the bounding box based on it. Can anyone help to explain where such spatial information comes from?