CUDA path seems to be hard-coded

My CUDA-9.0 is installed at /scratch_net/hox/mhahner/apps/cuda-9.0 because I have no write permissions at /usr/local

Where is the CUDA path assumed? How can I fix this?

The script already found my CUDA installation path correctly, see – Found CUDA: /scratch_net/hox/mhahner/apps/cuda-9.0 (found version “9.0”) but still raises the error: make[2]: *** No rule to make target ‘/usr/local/cuda/lib64/libnvToolsExt.so’, needed by ‘…/lib.linux-x86_64-3.7/spconv/libspconv.so’. Stop.

Here are the full details:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 9.9 (stretch)
Release:	9.9
Codename:	stretch

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176

$ cat $CUDA_ROOT/include/cudnn.h | grep CUDNN_MAJOR -A 2
#define CUDNN_MAJOR 7
#define CUDNN_MINOR 1
#define CUDNN_PATCHLEVEL 3

$ cmake --version
cmake version 3.15.2

$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  390.116  Sun Jan 27 07:21:36 PST 2019
GCC version:  gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)

$ python
Python 3.7.4 (default, Aug 13 2019, 20:35:49) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print(torch.__version__)
1.2.0

$ echo $CUDA_ROOT
/scratch_net/hox/mhahner/apps/cuda-9.0

$ echo $CUDA_TOOLKIT_ROOT
/scratch_net/hox/mhahner/apps/cuda-9.0/lib64

$ echo $CMAKE_CUDA_COMPILER
/scratch_net/hox/mhahner/apps/cuda-9.0/bin/nvcc

$ python setup.py bdist_wheel
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/spconv
copying spconv/functional.py -> build/lib.linux-x86_64-3.7/spconv
copying spconv/ops.py -> build/lib.linux-x86_64-3.7/spconv
copying spconv/pool.py -> build/lib.linux-x86_64-3.7/spconv
copying spconv/modules.py -> build/lib.linux-x86_64-3.7/spconv
copying spconv/conv.py -> build/lib.linux-x86_64-3.7/spconv
copying spconv/test_utils.py -> build/lib.linux-x86_64-3.7/spconv
copying spconv/__init__.py -> build/lib.linux-x86_64-3.7/spconv
creating build/lib.linux-x86_64-3.7/spconv/utils
copying spconv/utils/__init__.py -> build/lib.linux-x86_64-3.7/spconv/utils
running build_ext
Release
|||||CMAKE ARGS||||| ['-DCMAKE_PREFIX_PATH=/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch', '-DPYBIND11_PYTHON_VERSION=3.7', '-DSPCONV_BuildTests=OFF', '-DSPCONV_BuildCUDA=OFF', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/scratch_net/hox/mhahner/polybox/PhD/repositories/spconv/build/lib.linux-x86_64-3.7/spconv', '-DCMAKE_BUILD_TYPE=Release']
-- The CXX compiler identification is GNU 6.3.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found CUDA: /scratch_net/hox/mhahner/apps/cuda-9.0 (found version "9.0") 
-- Caffe2: CUDA detected: 9.0
-- Caffe2: CUDA nvcc is: /scratch_net/hox/mhahner/apps/cuda-9.0/bin/nvcc
-- Caffe2: CUDA toolkit directory: /scratch_net/hox/mhahner/apps/cuda-9.0
-- Caffe2: Header version is: 9.0
-- Found CUDNN: /scratch_net/hox/mhahner/apps/cuda-9.0/include  
-- Found cuDNN: v7.1.3  (include: /scratch_net/hox/mhahner/apps/cuda-9.0/include, library: /scratch_net/hox/mhahner/apps/cuda-9.0/lib64/libcudnn.so)
-- Autodetected CUDA architecture(s):  3.5
-- Added CUDA NVCC flags for: -gencode;arch=compute_35,code=sm_35
-- Found torch: /home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so  
-- Found PythonInterp: /home/mhahner/scratch/apps/anaconda3/envs/pytorch/bin/python3.7 (found suitable version "3.7.4", minimum required is "3.7") 
-- Found PythonLibs: /home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/libpython3.7m.so
-- Performing Test HAS_CPP14_FLAG
-- Performing Test HAS_CPP14_FLAG - Success
-- pybind11 v2.3.dev0
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Configuring done
-- Generating done
-- Build files have been written to: /scratch_net/hox/mhahner/polybox/PhD/repositories/spconv/build/temp.linux-x86_64-3.7
Scanning dependencies of target spconv_utils
Scanning dependencies of target spconv
[ 12%] Building CXX object src/utils/CMakeFiles/spconv_utils.dir/all.cc.o
[ 25%] Building CXX object src/spconv/CMakeFiles/spconv.dir/all.cc.o
[ 37%] Building CXX object src/spconv/CMakeFiles/spconv.dir/indice.cc.o
[ 50%] Building CXX object src/spconv/CMakeFiles/spconv.dir/reordering.cc.o
[ 62%] Building CXX object src/spconv/CMakeFiles/spconv.dir/maxpool.cc.o
[ 75%] Building CXX object src/spconv/CMakeFiles/spconv.dir/nms.cc.o
make[2]: *** No rule to make target '/usr/local/cuda/lib64/libnvToolsExt.so', needed by '../lib.linux-x86_64-3.7/spconv/libspconv.so'.  Stop.
make[2]: *** Waiting for unfinished jobs....
[ 87%] Linking CXX shared library ../../../lib.linux-x86_64-3.7/spconv/spconv_utils.cpython-37m-x86_64-linux-gnu.so
[ 87%] Built target spconv_utils
CMakeFiles/Makefile2:117: recipe for target 'src/spconv/CMakeFiles/spconv.dir/all' failed
make[1]: *** [src/spconv/CMakeFiles/spconv.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2
Traceback (most recent call last):
  File "setup.py", line 99, in <module>
    zip_safe=False,
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 192, in run
    self.run_command('build')
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "setup.py", line 40, in run
    self.build_extension(ext)
  File "setup.py", line 83, in build_extension
    subprocess.check_call(['cmake', '--build', '.'] + build_args, cwd=self.build_temp)
  File "/home/mhahner/scratch/apps/anaconda3/envs/pytorch/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j4']' returned non-zero exit status 2.

[MartinHahner]

After almost an entire day of debugging I finally managed to build spconv.

In order to successfully build spconv I had to modify setup.py in the following way

Before:

subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
subprocess.check_call(['cmake', '--build', '.'] + build_args, cwd=self.build_temp)

After

subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)

build_make_file = 'build/temp.linux-x86_64-3.7/src/spconv/CMakeFiles/spconv.dir/build.make'
link_file = 'build/temp.linux-x86_64-3.7/src/spconv/CMakeFiles/spconv.dir/link.txt'

for file in [build_make_file, link_file]:

    with open(file) as f:
        newText = f.read().replace('/usr/local/cuda', env['CUDA_ROOT'])

    with open(file, "w") as f:
        f.write(newText)

subprocess.check_call(['cmake', '--build', '.'] + build_args, cwd=self.build_temp)

[MartinHahner]

In case anyone wants to build spconv v.1.0 (commit 8da6f96) with a fairly up-to-date conda environment, here is how we in our lab were able to do it:

• download and unzip our hotfixes

• conda create --name spconv python=3.7 pytorch=1.4 cudatoolkit=10.1 cudatoolkit-dev=10.1 cmake --channel pytorch

• conda activate spconv

• conda install cudnn

• git clone https://github.com/traveller59/spconv --recursive

• cd spconv

• git checkout 8da6f967fb9a054d8870c3515b1b44eca2103634

• git am <path_to_hotfixes>/0001-fix-problem-with-torch-1.4.patch

• git am <path_to_hotfixes>/0001-Allow-to-specifiy-CUDA_ROOT-directory-and-pick-corre.patch

• CUDA_ROOT=<path_to_your_conda_installation>/envs/spconv python setup.py bdist_wheel

• cd dist/

• pip install *

Thank me later.