Inconsistent model inference

Describe the bug

Outputs different predictions for the same input depending batch. I suspect the problem is related to pad tokens not being masked properly.

Steps/Code to reproduce bug

import nemo.collections.asr as nemo_asr

model = nemo_asr.models.ASRModel.from_pretrained(model_name="stt_en_conformer_ctc_large")
model.transcribe(audio_paths[:2], batch_size=1)

# output
# ['take care of over excitement and endeavor to keep a quiet mind even for your health it is the best advice that can be given to you # # your moral and spiritual improvement will then keep pace with the culture of your intellectual powers',
# 'ab crowd']

model.transcribe(audio_paths[:2], batch_size=2)
# output
# ['take care of over excitement and endeavor to keep a quiet mind even for your health it is the best advice that can be given to you # # your moral and spiritual improvement will then keep pace with the culture of your intellectual powers',
# 'crow']

# the provided example occurred with audio files from the libritts100 dataset namely (3575_170457_000032_000001.wav and 6829_68771_000042_000001.wav), but I am quite sure it can be reproduced with other audio files as well. 

Expected behavior

Outputs should be identical independent of batch size or items in batch.

Environment overview

Linux Ubuntu 20.04 Python version: 3.8 PyTorch version: 1.10.0+cu113 Nemo version: 1.4.0 CUDA/cuDNN version: 11 GPU model and memory: RTX 3070, Tesla V100-SXM2-16GB

[titu1994]

It would take two pases over the audio to measure just the durations then sort and infer. This doesn’t scale for large datasets or multi node inference (parallel transcribe script).

We use batch size 1 for academic settings - when publishing numbers for a paper cause it’s a one off anyway. For all other settings we use batch size 32 or more (for RNNTs as high as 256) and accept a 0.1% loss of WER.

[sipan17]

I measured on 43k audios of libritts100 dataset, it took 3 minutes with 8 processes, which might be preferred to error coming from padding. Anyway, it’s just a solution that came to my mind.