How to use GPU during inference?

The example code for inference doesn’t seem to be using GPU. How can I do so?

I’ve tried this but getting a CUDNN_STATUS_NOT_INITIALIZED error:

run_opts = {"device": "cuda","data_parallel_count": -1,"data_parallel_backend": False,"distributed_launch": False,"distributed_backend": "nccl","jit_module_keys": None}

asr_model = EncoderDecoderASR.from_hparams(source="speechbrain/asr-crdnn-rnnlm-librispeech", savedir="pretrained_model", run_opts=run_opts)
```

[pplantinga]

Try:

asr_model = TransformerASR.from_hparams(
  source="speechbrain/asr-transformer-transformerlm-librispeech",
  savedir="pretrained_models/asr-transformer-transformerlm-librispeech",
  run_opts={"device": "cuda"},
)

edit: as pointed out by @mschonwe changed last } to )

[alpoktem]

I was wrong trusting torch.cuda.is_available() saying everything is fine. I had upgraded to PyTorch 1.8 as required by speechbrain and it broke my other GPU training setups too. I was getting this error:

`RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasCreate(handle)`

Reverting back into torch 1.7.1 fixed it for me. I am now able to run both my and @mschonwe’s code using GPU. Thanks for the guidance @pplantinga.