Some questions about dropout

Hi again @lucidrains, I just had some quick questions about dropout with the Sinkhorn Transformer, as I was just using my Linformer implementation (which as you know is based off of this repo), but it was overfitting my dataset. Therefore, I just had some quick questions about some dropout and your implementation, and I wanted to ask whether some design choices here were intentional or not:

1. In the original Transformer, dropout was performed after each sublayer, before the residual connection. I noticed that you only have this after the SinkhornSelfAttention class, but not after the FeedForward class. Is this intentional?
2. Speaking of the FeedForward class, you insert dropout after the first linear layer. I couldn’t find this anywhere in any literature, were you able to find a reference of why this was effective? I put it into my implementation, and it seems to help, but i just don’t know where this idea came from.
3. On a similar note, do you know why the dots tensor in the self attention classes are dropped out? Again, I put it in my linformer and it seems to work, but I can’t find a reference to this in the literature.
4. Finally, the original transformer also dropped out the input tokens, like so (From the SinkhornTransformerLM class):

    def forward(self, x, **kwargs):
        _, t, device = *x.shape, x.device
        assert t <= self.max_seq_len, f'sequence length {t} is greater than maximum sequence length {self.max_seq_len}'

        x = self.to_token_emb(x)
        x = self.axial_pos_emb(x) + x
        """ Dropout would go here"""
        x = self.sinkhorn_transformer(x, **kwargs)
        return self.to_logits(x)

Should they also be dropped out here as well?

I now updated my repo such that all 4 of these dropout possibilities exist. I’ll let you know if this helps overfitting.

Thank you for your time!

[lucidrains]

@tatp22 Nice! Thank you for sharing your experience 😃

[tatp22]

@lucidrains Right now, I’m only working on seq lengths of 2048, and I am planning to scale it to 8096 soonish. To be honest, I did try a (practical) sequence length of 250k+ (with a k of 150), and it did end up competing with baselines, but I am not pursuing these experiments further atm.

From what I have experienced though, practically, the Linformer works very well (with respect to the standard transformer), even if the k is very small. However, one thing that one must watch out for is that the parameter numbers can seriously explode, especially with longer sequences (compared to standard attention, and even with this repo).

Personally, I have a feeling that attention in general does not need to be quadratic (in time and space), and there may just be better architectures that one can use that are faster and more memory efficient. Unfortunately, I am not really in a position to investigate this at the moment, due to time limitations