The sequence of operations in the Linformer Attention module is probably wrong.

🐛 Bug

Hi guys! In Linformer’s example source code, I found that operation order may not match the official paper mathematics.

Here, in the code, the linear attention is done in the following sequence of the two operations:

1. a Linear projection from n token’s representations to k.
2. a Linear projection over the embedding dimension (d_m to d_k). as here (#208 and #213 respectively)

On the contrary, this image from the Linformer paper states that it should be performed in the order of:

1. a Linear projection over the embedding dimension (d_m to d_k).
2. a Linear projection from n token’s representations to k. As seen in the picture below:

Am I missing something important here? If anything gets confirmed, I am up for fixing it.

Environment

Current fairseq version.

[patil-suraj]

Any update on this?

[myleott]

@madian9