Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

WSL cuda 11.2: CUDARuntimeError: cudaErrorInvalidValue: invalid argument

See original GitHub issue

I installed cupy on WSL, and get a runtime error when trying to use cupy.

Kernel

» uname -a
Linux DESKTOP-1OG5J92 4.19.128-microsoft-standard #1 SMP Tue Jun 23 12:58:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
» cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"

Cupy

» python -c 'import cupy; cupy.show_config()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/plate/.local/lib/python3.8/site-packages/cupy/__init__.py", line 866, in show_config
    _sys.stdout.write(str(_cupyx.get_runtime_info()))
  File "/home/plate/.local/lib/python3.8/site-packages/cupyx/_runtime.py", line 240, in get_runtime_info
    return _RuntimeInfo()
  File "/home/plate/.local/lib/python3.8/site-packages/cupyx/_runtime.py", line 138, in __init__
    self.cublas_version = _eval_or_error(
  File "/home/plate/.local/lib/python3.8/site-packages/cupyx/_runtime.py", line 49, in _eval_or_error
    return func()
  File "/home/plate/.local/lib/python3.8/site-packages/cupyx/_runtime.py", line 140, in <lambda>
    cupy.cuda.device.get_cublas_handle()),
  File "cupy/cuda/device.pyx", line 46, in cupy.cuda.device.get_cublas_handle
  File "cupy/cuda/device.pyx", line 47, in cupy.cuda.device.get_cublas_handle
  File "cupy/cuda/device.pyx", line 29, in cupy.cuda.device._get_device
  File "cupy_backends/cuda/api/runtime.pyx", line 275, in cupy_backends.cuda.api.runtime.getDevice
  File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorInvalidValue: invalid argument

CUDA

» nvidia-smi.exe
Fri Apr  9 13:13:07 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.12       Driver Version: 465.12       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 165... WDDM  | 00000000:02:00.0 Off |                  N/A |
| N/A   40C    P8     3W /  N/A |    132MiB /  4096MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+


+-----------------------------------------------------------------------------+
| Processes:
 |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found
 |
 
 » apt list --installed | grep cuda

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

cuda-command-line-tools-11-2/unknown,unknown,now 11.2.2-1 amd64 [installed,automatic]
cuda-compiler-11-2/unknown,unknown,now 11.2.2-1 amd64 [installed,automatic]
cuda-cudart-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-cudart-dev-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-cuobjdump-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-cupti-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-cupti-dev-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-cuxxfilt-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-documentation-11-2/unknown,unknown,now 11.2.154-1 amd64 [installed,automatic]
cuda-driver-dev-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-gdb-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-libraries-11-2/unknown,unknown,now 11.2.2-1 amd64 [installed,automatic]
cuda-libraries-dev-11-2/unknown,unknown,now 11.2.2-1 amd64 [installed,automatic]
cuda-memcheck-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-nsight-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-nsight-compute-11-2/unknown,unknown,now 11.2.2-1 amd64 [installed,automatic]
cuda-nsight-systems-11-2/unknown,unknown,now 11.2.2-1 amd64 [installed,automatic]
cuda-nvcc-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-nvdisasm-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-nvml-dev-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-nvprof-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-nvprune-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-nvrtc-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-nvrtc-dev-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-nvtx-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-nvvp-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-samples-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-sanitizer-11-2/unknown,unknown,now 11.2.152-1 amd64 [installed,automatic]
cuda-toolkit-11-2/unknown,unknown,now 11.2.2-1 amd64 [installed]
cuda-tools-11-2/unknown,unknown,now 11.2.2-1 amd64 [installed,automatic]
cuda-visual-tools-11-2/unknown,unknown,now 11.2.2-1 amd64 [installed,automatic]

I also verified my environment using pytorch, and it looks like it’s working, torch.cuda.is_available() is True