RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at /home/username/github/maskrcnn-benchmark/maskrcnn_benchmark/csrc/cuda/nms.cu:103

❓RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at /home/username/github/maskrcnn-benchmark/maskrcnn_benchmark/csrc/cuda/nms.cu:103

Hello, when I run the Mask_R-CNN_demo.ipynb, in the ‘cuda’ mode, everything goes well until the last kernel. When running predictions = coco_demo.run_on_opencv_image(image), the error happens.

For finding the reason, I try the following things, which make me more confused: As an output of collect_env.py, I get the following informations:

Collecting environment information...
PyTorch version: 1.0.0.dev20181106
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 18.04.1 LTS
GCC version: (Ubuntu 6.4.0-17ubuntu1) 6.4.0 20180424
CMake version: version 3.10.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 9.1.85
GPU models and configuration: 
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti

Nvidia driver version: 390.67
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.0.5
/usr/lib/x86_64-linux-gnu/libcudnn_static_v7.a

Versions of relevant libraries:
[pip3] numpy (1.15.1)
[pip3] torch (0.4.1)
[conda] Could not collect

And I also try to run the ./deviceQuery in NVIDIA_CUDA-9.1_Samples/bin/x86_64/linux/release, it gives the PASS result, more concretely:

deviceQuery, 
CUDA Driver = CUDART, 
CUDA Driver Version = 9.1, 
CUDA Runtime Version = 9.1, 
NumDevs = 2
Result = PASS

All the tests above shows that the cuda runtime version matches the cuda driver version. I hence don’t know where comes from this runtime error, and how to fix it?

Thank you.

[lanpa]

Have you tried removing all torch related libs and then reinstall with conda install pytorch-nightly cuda92 -c pytorch ?

[fmassa]

I’l try doing what @lanpa mentioned. I don’t really have any other suggestions 😕