Issue setting up GPU

Good day!

Thanks for developing this great tool. I have been trying to set up my environment to run using GPU to speed up the computation but have not succeeded. I have two types of errors when using different graphic cards.

Initially, with a TeslaV100, I Had the next error:

Can not use cuDNN on context None: cannot compile with cuDNN. We got this error:
b'/scratch/4102320/try_flags_92ylr6tr.c:4:19: fatal error: cudnn.h: No such file or directory\n #include <cudnn.h>\n                   ^\ncompilation terminated.\n'
Mapped name None to device cuda0: Tesla V100-SXM2-16GB (0000:04:00.0)

I thought It was because the conda env lacked the cudnn package. I installed it and the error is now different:

Can not use cuDNN on context None: cannot compile with cuDNN. We got this error:
b'/scratch/4102320/try_flags_92ylr6tr.c:4:19: fatal error: cudnn.h: No such file or directory\n #include <cudnn.h>\n                   ^\ncompilation terminated.\n'
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "/hpc/pmc_stunnenberg/cruiz/miniconda3/envs/python_pHGG_project/lib/python3.7/site-packages/theano/gpuarray/__init__.py", line 227, in <module>
    use(config.device)
  File "/hpc/pmc_stunnenberg/cruiz/miniconda3/envs/python_pHGG_project/lib/python3.7/site-packages/theano/gpuarray/__init__.py", line 214, in use
    init_dev(device, preallocate=preallocate)
  File "/hpc/pmc_stunnenberg/cruiz/miniconda3/envs/python_pHGG_project/lib/python3.7/site-packages/theano/gpuarray/__init__.py", line 159, in init_dev
    pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True)
  File "pygpu/blas.pyx", line 149, in pygpu.blas.gemm
  File "pygpu/blas.pyx", line 47, in pygpu.blas.pygpu_blas_rgemm
pygpu.gpuarray.GpuArrayException: (b'cublasCreate: (cublas) Library not initialized. (Possibly because the driver version is too old for the cuda version)', 11)

However, if I used other GPU, (GeForce RTX2080i, Quadro RTX 6000), the error is the same than the first for the Tela GPU

Can not use cuDNN on context None: cannot compile with cuDNN. We got this error:
b'/scratch/4103662/try_flags_q9s6i39u.c:4:19: fatal error: cudnn.h: No such file or directory\n #include <cudnn.h>\n                   ^\ncompilation terminated.\n'
Mapped name None to device cuda: Quadro RTX 6000 (0000:86:00.0)

Info of my packages/modules:

sys 3.7.10 | packaged by conda-forge | (default, Feb 19 2021, 16:07:37) 
[GCC 9.3.0]
ipykernel 5.5.3
ipykernel._version 5.5.3
json 2.0.9
re 2.2.1
IPython 7.22.0
IPython.core.release 7.22.0
logging 0.5.1.2
zlib 1.0
traitlets 5.0.5
traitlets._version 5.0.5
argparse 1.1
ipython_genutils 0.2.0
ipython_genutils._version 0.2.0
platform 1.0.8
IPython.core.crashhandler 7.22.0
pygments 2.8.1
pexpect 4.8.0
ptyprocess 0.7.0
decorator 5.0.5
pickleshare 0.7.5
backcall 0.2.0
sqlite3 2.6.0
sqlite3.dbapi2 2.6.0
_sqlite3 2.6.0
prompt_toolkit 3.0.18
wcwidth 0.2.5
jedi 0.18.0
parso 0.8.2
colorama 0.4.4
ctypes 1.1.0
_ctypes 1.1.0
IPython.core.magics.code 7.22.0
urllib.request 3.7
jupyter_client 6.1.12
jupyter_client._version 6.1.12
zmq 22.0.3
zmq.backend.cython 40304
zmq.backend.cython.constants 40304
zmq.sugar 22.0.3
zmq.sugar.constants 40304
zmq.sugar.version 22.0.3
jupyter_core 4.7.1
jupyter_core.version 4.7.1
tornado 6.1
_curses b'2.2'
dateutil 2.8.1
dateutil._version 2.8.1
six 1.15.0
decimal 1.70
_decimal 1.70
distutils 3.7.10
scanpy 1.7.1
scanpy._metadata 1.7.1
packaging 20.9
packaging.__about__ 20.9
pkg_resources._vendor.six 1.10.0
pkg_resources.extern.six 1.10.0
pkg_resources._vendor.appdirs 1.4.3
pkg_resources.extern.appdirs 1.4.3
pkg_resources._vendor.packaging 20.4
pkg_resources._vendor.packaging.__about__ 20.4
pkg_resources.extern.packaging 20.4
pkg_resources._vendor.pyparsing 2.2.1
pkg_resources.extern.pyparsing 2.2.1
csv 1.0
_csv 1.0
numpy 1.20.2
numpy.version 1.20.2
numpy.core 1.20.2
numpy.core._multiarray_umath 3.1
numpy.lib 1.20.2
numpy.linalg._umath_linalg 0.1.5
scipy 1.6.2
scipy.version 1.6.2
anndata 0.7.5
anndata._metadata 0.7.5
h5py 3.1.0
h5py.version 3.1.0
cached_property 1.5.2
natsort 7.1.1
pandas 1.2.3
pytz 2021.1
pandas.compat.numpy.function 1.20.2
zarr 2.7.0
numcodecs 0.7.3
numcodecs.version 0.7.3
numcodecs.blosc 1.18.1
numcodecs.zstd 1.4.4
numcodecs.lz4 1.9.2
zarr.version 2.7.0
dask 2021.04.0
yaml 5.4.1
tlz 0.11.1
toolz 0.11.1
psutil 5.8.0
cloudpickle 1.6.0
fsspec 0.9.0
scipy._lib._uarray 0.5.1+49.g4c3f1d7.scipy
sinfo 0.3.1
stdlib_list v0.7.0
numba 0.53.1
llvmlite 0.36.0
numba.misc.appdirs 1.4.1
sklearn 0.24.1
sklearn.base 0.24.1
joblib 1.0.1
joblib.externals.loky 2.9.0
joblib.externals.cloudpickle 1.6.0
scipy._lib.decorator 4.0.5
scipy.linalg._fblas b'$Revision: $'
scipy.linalg._flapack b'$Revision: $'
scipy.linalg._flinalg b'$Revision: $'
scipy.special.specfun b'$Revision: $'
scipy.ndimage 2.0
scipy.optimize.minpack2 b'$Revision: $'
scipy.sparse.linalg.isolve._iterative b'$Revision: $'
scipy.sparse.linalg.eigen.arpack._arpack b'$Revision: $'
scipy.optimize._lbfgsb b'$Revision: $'
scipy.optimize._cobyla b'$Revision: $'
scipy.optimize._slsqp b'$Revision: $'
scipy.optimize._minpack  1.10 
scipy.optimize.__nnls b'$Revision: $'
scipy.linalg._interpolative b'$Revision: $'
scipy.integrate._odepack  1.9 
scipy.integrate._quadpack  1.13 
scipy.integrate._ode $Id$
scipy.integrate.vode b'$Revision: $'
scipy.integrate._dop b'$Revision: $'
scipy.integrate.lsoda b'$Revision: $'
scipy.interpolate._fitpack  1.7 
scipy.interpolate.dfitpack b'$Revision: $'
scipy.stats.statlib b'$Revision: $'
scipy.stats.mvn b'$Revision: $'
sklearn.utils._joblib 1.0.1
leidenalg 0.8.3
igraph 0.9.1
texttable 1.6.3
igraph.version 0.9.1
louvain 0.7.0
matplotlib 3.4.1
PIL 8.1.2
PIL._version 8.1.2
PIL.Image 8.1.2
xml.etree.ElementTree 1.3.0
cffi 1.14.5
pyparsing 2.4.7
cycler 0.10.0
kiwisolver 1.3.1
tables 3.6.1
numexpr 2.7.3
numexpr.version 2.7.3
legacy_api_wrap 0.0.0
get_version 2.1
umap 0.5.1
_cffi_backend 1.14.5
pycparser 2.20
pycparser.ply 3.9
pycparser.ply.yacc 3.10
pycparser.ply.lex 3.10
pynndescent 0.5.2
theano 1.0.5
theano.version 1.0.5
mkl 2.3.0
scipy.signal.spline 0.2
pygpu 0.7.6
mako 1.1.4
markupsafe 1.1.1
plotnine 0.7.0
patsy 0.5.1
patsy.version 0.5.1
mizani 0.7.3
palettable 3.3.0
mizani.external.husl 4.0.3
statsmodels 0.12.2
statsmodels.api 0.12.2
statsmodels.__init__ 0.12.2
statsmodels.tools.web 0.12.2
pymc3 3.9.3
xarray 0.17.0
netCDF4 1.5.6
netCDF4._netCDF4 1.5.6
cftime 1.4.1
cftime._cftime 1.4.1
arviz 0.10.0
arviz.data.base 0.10.0
fastprogress 0.2.7
tqdm 4.59.0
tqdm.cli 4.59.0
tqdm.version 4.59.0
tqdm._dist_ver 4.59.0
ipywidgets 7.6.3
ipywidgets._version 7.6.3
torch 1.8.1+cu102
torch.version 1.8.1+cu102
tarfile 0.9.0
torch.cuda.nccl 2708
torch.backends.cudnn 7605
seaborn 0.11.1
seaborn.external.husl 2.1.0

Do you know what the issue might be?

Thanks in advance for your help!

[vitkl]

Also added a note to common issues: https://github.com/BayraktarLab/cell2location/blob/master/README.md#3-theano-fails-to-use-the-gpu-at-all-or-cudnn-in-particular

[ccruizm]

I created the conda env with the environment.yml but later installed cudnn. I will create a fresh environment and test it again.

The error using singularity and TeslaV100 was the following:

/opt/conda/envs/cellpymc/lib/python3.7/site-packages/theano/gpuarray/dnn.py:184: UserWarning: Your cuDNN version is more recent than Theano. If you encounter problems, try updating Theano or downgrading cuDNN to a version >= v5 and <= v7.
  warnings.warn("Your cuDNN version is more recent than "
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "/opt/conda/envs/cellpymc/lib/python3.7/site-packages/theano/gpuarray/__init__.py", line 227, in <module>
    use(config.device)
  File "/opt/conda/envs/cellpymc/lib/python3.7/site-packages/theano/gpuarray/__init__.py", line 214, in use
    init_dev(device, preallocate=preallocate)
  File "/opt/conda/envs/cellpymc/lib/python3.7/site-packages/theano/gpuarray/__init__.py", line 117, in init_dev
    context.cudnn_handle = dnn._make_handle(context)
  File "/opt/conda/envs/cellpymc/lib/python3.7/site-packages/theano/gpuarray/dnn.py", line 130, in _make_handle
    "This can be a sign of a too old driver.", err)
RuntimeError: ('Error creating cudnn handle. This can be a sign of a too old driver.', 1)