Calling cupy.linalg.svd results in the error CUSOLVER_STATUS_NOT_INITIALIZED

Description

I installed cudatoolkit as:

conda install -c nvidia cuda
conda install -c conda-forge cupy

I have 8 Quadro RTX 5000 GPUs, and my nvidia-smi output is: NVIDIA-SMI 520.61.05 Driver Version: 520.61.05 CUDA Version: 11.8

My cudatoolkit is 11.8.0, build h37601d7_10 from the fastchan repo My cupy is 9.6.0, build py39ha91d172_0, from the conda-forge repository

To Reproduce

import cupy as cp
#Trying to fit a circle from points
xm = cp.zeros(shape=(256, 256), dtype=cp.float64)
ym = cp.copy(Xm)
Rm = ((xm**2) + (ym**2))**0.5
radius = 50
Ym, Xm = cp.mgrid[0:Rm.shape[0]. 0:Rm.shape[1]]
Ym = Ym[Rm==radius]
Xm = Xm[Rm==radius]
Am = cp.hstack([Xm**2, Xm * Ym, Ym**2])
Bm = cp.ones_like(Xm)
a, b, c = cp.linalg.lstsq(Am, Bm, rcond=None)[0].squeeze()

Installation

Conda-Forge (conda install ...)

Environment

---------------------------------------------------------------------------
CUSOLVERError                             Traceback (most recent call last)
Cell In[21], line 3
      1 Am = cp.hstack([Xm**2, Xm * Ym, Ym**2])
      2 Bm = cp.ones_like(Xm)
----> 3 a, b, c = cp.linalg.lstsq(Am, Bm, rcond=None)[0].squeeze()

File ~/anaconda3/envs/intersect/lib/python3.9/site-packages/cupy/linalg/_solve.py:258, in lstsq(a, b, rcond)
    255 if m != m2:
    256     raise linalg.LinAlgError('Incompatible dimensions')
--> 258 u, s, vh = cupy.linalg.svd(a, full_matrices=False)
    260 if rcond is None:
    261     rcond = numpy.finfo(s.dtype).eps * max(m, n)

File ~/anaconda3/envs/intersect/lib/python3.9/site-packages/cupy/linalg/_decomposition.py:503, in svd(a, full_matrices, compute_uv)
    501     job_vt = ord('N')
    502 s = cupy.empty(k, dtype=real_dtype)
--> 503 handle = device.get_cusolver_handle()
    504 dev_info = cupy.empty(1, dtype=numpy.int32)
    506 if dtype == 'f':

File cupy/cuda/device.pyx:72, in cupy.cuda.device.get_cusolver_handle()

File cupy/cuda/device.pyx:73, in cupy.cuda.device.get_cusolver_handle()

File cupy/cuda/device.pyx:238, in cupy.cuda.device.Device.cusolver_handle.__get__()

File cupy/cuda/device.pyx:214, in cupy.cuda.device.Device._get_handle()

File cupy/cuda/device.pyx:215, in cupy.cuda.device.Device._get_handle()

File cupy_backends/cuda/libs/cusolver.pyx:1077, in cupy_backends.cuda.libs.cusolver.create()

File cupy_backends/cuda/libs/cusolver.pyx:1081, in cupy_backends.cuda.libs.cusolver.create()

File cupy_backends/cuda/libs/cusolver.pyx:1052, in cupy_backends.cuda.libs.cusolver.check_status()

CUSOLVERError: CUSOLVER_STATUS_NOT_INITIALIZED

Additional Information

No response

[leofang]

conda-forge packages are meant to be not mix-n-match’d with other channels:

conda create -n my_new_env -c conda-forge -y cupy
conda activate my_new_env

[leofang]

Great! How about CuPy v11 if you can upgrade? 😛