Strange behavior using PyTorch DDP

@1ytic Hi,

So far I have been able to use the loss with DDP on a single GPU , it behaves more or less as expected.

But when I use more than 1 device, the following happens:

• On GPU-0 loss is calculated properly
• On GPU-1 loss is close to zero for each batch

I checked the input tensors, devices, tensor values, etc - so far everything seems to be identical for GPU-0 and other GPUs.

[csukuangfj]

Thanks for the heads up about the torchaudio loss!

@snakers4 You may find https://github.com/danpovey/fast_rnnt useful.

[burchim]

Yes, this means that logits / target lengths tensors do not match the logits / target tensors. If you have logits lengths longer than your logits tensor for instance.