Why inplace operation in `AdaptiveSoftmax`

I used log_output for training purpose, but pytorch complains about inplace operation on backward.

For lines 204 and 205: https://github.com/pytorch/fairseq/blob/e6422528dae0b899848469efe2dc404c1e639ce9/fairseq/modules/adaptive_softmax.py#L200-L208

1. why doing a copy_ on tailout
2. why inplace add_

I changed to

tail_output = ...
... = self.lsm(tail_out) + tail_priors[idxs, i, None]

And it works for training.

[Stonesjtu]

1. The current implementation is not working in training mode.
2. In evaluation mode, if we set the torch.no_grad env, the memory overhead should equal to the tail_priors[idxs, i ,None] (temporarily for the out-of-place add), (edit: plus the size of log_probs[idxs, start:end] for re-use as storage of self.tail[i](input[idxs])).

Actually add operation does not require saving operands for future BP, but pytorch does raise an error for any inplace operators. The reason is that using in-place operation is quite dangerous for chain-rule BP.

I recommend changing to an out-of-place version, I can open a PR and give a simple comparison in terms of memory.

If the memory saving is very critical then I would suggest implementing a Function to by-pass the in-place check mechanism.

[stale[bot]]

Closing this issue after a prolonged period of inactivity. If this issue is still present in the latest release, please create a new issue with up-to-date information. Thank you!