About the Duplex attention

Hi, Thanks for sharing the code!

I have a few questions about Section 3.1.2. Duplex attention.

1. I am confused by the notation in the section. For example, in this section, “Y=(K^{P\times d}, V^{P\times d}), where the values store the content of the Y variables (e.g. the randomly sampled latents for the case of GAN)”. Does it mean that V^{P\times d} is sampled from the original variable Y? how to set the number of P in your code?

2. “keys track the centroids of the attention-based assignments from X to Y, which can be computed as K=a_b(Y, X)”, does it mean K is calculated by using the self-attention module but with (Y, X) as input? If so, how to understand “the keys track the centroid of the attention-based assignments from X to Y”? BTW, how to get the centroids?

3. For the update rule in duplex attention, what does the a() function mean? Does it denote a self-attention module like a_b() in Section 3.1.1, where X as query, K as keys, and V as values, if so, K is calculated from another self-attention module as mentioned in question 2, so the output of a_b(Y, X) will be treated as Keys, so the update rule contains two self-attention operations? is that right? Does it mean ’Duplex‘ attention?

4. But finally I find I may be wrong when I read the last paragraph in this section. As mentioned in this section, “to support bidirectional interaction between elements, we can chain two reciprocal simplex attentions from X to Y and from Y to X, obtaining the duplex attention” So, does it mean, first, we calculate the Y by using a simplex attention module u^a(Y, X), and then use this Y as input of u^d(X, Y) to update X? Does it mean the duplex attention module contains three self-attention operations?

Thanks a lot! 😃

[AndrewChiyz]

Thanks a lot for your detailed reply! Now I understand the core idea of the duplex attention part.

Thank you! 😃

[dorarad]

Hi all! @07hyx06 Yep that’s correct! We first find the centroids by casting attention over the image features (x) and then update the features based on the centroids (K) @nicolas-dufour that’s right the values are not iteratively updated, only the centroids and the image features! @subminu Thanks so much for pointing that out! I’ll update the paper with that fix!