Extremely slow when evaluating

Hi, I am running stage 3 of egs/librimix/ConvTasNet/run.sh. I used --compute_wer 1 --eval_mode max to evaluate WER. However, it is running extremely slow.

2%|█▉                                                                                                  | 58/3000 [46:02<29:01:05, 35.51s/it

It takes more than one day to complete. I checked with nvidia-smi, and it was computing with GPU. However, I think only the separation process is running with GPU. I looked through the code eval.py and found out that numpy arrays are fed to the wer_tracker. So I think that for the asr part, it is evaluating in CPU mode. Is there any reason this can’t be calculated with GPUs?

By the way, I see that eval.py is evaluating with the “Shinji Watanabe/librispeech_asr_train_asr_transformer_e18_raw_bpe_sp_valid.acc.best” asr model. Is it possible to switch to other kinds of asr models by modifying line 52?

Thanks

[popcornell]

At the time full decoding on GPU was not implemented with batched inputs. I recall the reason was something like this, we had problems running on GPU. Looks like now it runs smooth on GPU, thanks to the ESPNet gang and thank you for trying this. If you have time add an argument like use_gpu and submit a PR it would be great

[mpariente]

Yes, ASR is very long on CPU. I don’t remember why we didn’t run it on GPU in the first place, maybe memory issues? Can’t remember. Maybe @popcornell or @JorisCos would remember?