How to drop samples larger than `batch_bins` when training (in order to avoid out of memory errors)?

Hello!

Is there a way to have batch_bins specify the maximum number of bins allowed by a batch (in the case of batches of type length or numel)? If I understand correctly, the current implementation groups the samples into a number of bins that are closest, but also greater than batch_bins; I would like (i) the batch size to be capped to at most batch_bins and (ii) if there are any samples larger than batch_bins to have them removed. The motivation is that there are cases when there are a few long audio files in the training dataset and those cause the entire process to run out of memory (especially when working on a small GPU). I assume that I can modify the corresponding batch samples (NumElementsBatchSampler and LengthBatchSampler), but I was wondering if there already exists solution to this type of problem.

Thank you!

[kan-bayashi]

For quick fixing, how about filtering in stage 4? https://github.com/espnet/espnet/blob/96e6eb97bacb428d9979661725657b5c38a17dee/egs2/TEMPLATE/asr1/asr.sh#L527

[kamo-naoyuki]

Thanks. I’ll consider it.

It doesn’t exist for sampler side now. If you’ll remove long samples, or samples having the other problems, from your training data, you can delete the lines of them from shape text directly.