Suspecious bug

I’ve noticed an abnormal phenomenon for seq2seq model: the validation loss is always lower than training loss, both on Twitter and OpenSubtitles dataset.

And I found this part of codes quite suspicious: https://github.com/facebookresearch/ParlAI/blob/09351eec1076319a9f5ea0f6a0c242d5f35fa43d/parlai/agents/seq2seq/seq2seq.py#L403-L411

So here we see when doing pure prediction, we don’t use true labels ys, but when computing loss we use, which I think will bias the validation loss. Are there any reasons for computing loss like this?

[ShaojieJiang]

And thanks @alexholdenmiller ! With dropout=0.0, the validation loss is roughly the same as training loss.

[alexholdenmiller]

Updated perplexity scoring