spconv v1.1 unknown device type

Did anyone run into this problem and was able to solve it?

It only occurs when training SECOND and PartA^2Net because they are utilizing spconv in their RPN backbone,
training PointPillars, on the other hand, works fine because it’s RPN (PointPillarsScatter) does not utilize spconv.

Is it a phenomenon only occurring with spconv v1.1 ?

Traceback (most recent call last):
  File "train.py", line 215, in <module>
    main()
  File "train.py", line 210, in main
    max_ckpt_save_num=arguments.max_ckpt_save_num)
  File "/scratch_net/hox/mhahner/repositories/PCDet/tools/train_utils/train_utils.py", line 80, in train_model
    leave_pbar=(cur_epoch + 1 == total_epochs)
  File "/scratch_net/hox/mhahner/repositories/PCDet/tools/train_utils/train_utils.py", line 36, in train_one_epoch
    loss, tb_dict, disp_dict = model_func(model, batch)
  File "/scratch_net/hox/mhahner/repositories/PCDet/pcdet/models/__init__.py", line 25, in model_func
    ret_dict, tb_dict, disp_dict = model(input_dict)
  File "/home/mhahner/scratch/apps/anaconda3/envs/apex/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/scratch_net/hox/mhahner/repositories/PCDet/pcdet/models/detectors/PartA2_net.py", line 106, in forward
    rpn_ret_dict = self.forward_rpn(**input_dict)
  File "/scratch_net/hox/mhahner/repositories/PCDet/pcdet/models/detectors/PartA2_net.py", line 33, in forward_rpn
    **kwargs
  File "/home/mhahner/scratch/apps/anaconda3/envs/apex/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/scratch_net/hox/mhahner/repositories/PCDet/pcdet/models/rpn/rpn_unet.py", line 471, in forward
    x = self.conv_input(input_sp_tensor)
  File "/home/mhahner/scratch/apps/anaconda3/envs/apex/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mhahner/scratch/apps/anaconda3/envs/apex/lib/python3.7/site-packages/spconv/modules.py", line 130, in forward
    input = module(input)
  File "/home/mhahner/scratch/apps/anaconda3/envs/apex/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mhahner/scratch/apps/anaconda3/envs/apex/lib/python3.7/site-packages/spconv/conv.py", line 177, in forward
    use_hash=self.use_hash)
  File "/home/mhahner/scratch/apps/anaconda3/envs/apex/lib/python3.7/site-packages/spconv/ops.py", line 93, in get_indice_pairs
    stride, padding, dilation, out_padding, int(subm), int(transpose), int(use_hash))
ValueError: /scratch_net/hox/mhahner/repositories/spconv/include/spconv/spconv_ops.h 132
false assert faild. unknown device type

# packages in environment at /home/mhahner/scratch/apps/anaconda3/envs/apex:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                      1_llvm    conda-forge
bzip2                     1.0.8                h516909a_2    conda-forge
ca-certificates           2020.1.1                      0
certifi                   2020.4.5.1               py37_0
cmake                     3.17.0               h28c56e5_0    conda-forge
coloredlogs               14.0                     pypi_0    pypi
cudatoolkit               10.1.243             h6bb024c_0
cudatoolkit-dev           10.1.243             h516909a_3    conda-forge
cudnn                     7.6.5                cuda10.1_0
cycler                    0.10.0                   pypi_0    pypi
decorator                 4.4.2                    pypi_0    pypi
easydict                  1.9                      pypi_0    pypi
expat                     2.2.9                he1b5a44_2    conda-forge
humanfriendly             8.2                      pypi_0    pypi
imagecodecs               2020.2.18                pypi_0    pypi
imageio                   2.8.0                    pypi_0    pypi
kiwisolver                1.2.0                    pypi_0    pypi
krb5                      1.17.1               h2fd8d38_0    conda-forge
libblas                   3.8.0               16_openblas    conda-forge
libcblas                  3.8.0               16_openblas    conda-forge
libcurl                   7.69.1               hf7181ac_0    conda-forge
libedit                   3.1.20181209         hc058e9b_0
libffi                    3.2.1             he1b5a44_1007    conda-forge
libgcc-ng                 9.2.0                h24d8f2e_2    conda-forge
libgfortran-ng            7.5.0                hdf63c60_6    conda-forge
liblapack                 3.8.0               16_openblas    conda-forge
libopenblas               0.3.9                h5ec1e0e_0    conda-forge
libssh2                   1.8.2                h22169c7_2    conda-forge
libstdcxx-ng              9.2.0                hdf63c60_2    conda-forge
libuv                     1.34.0               h516909a_0    conda-forge
llvm-openmp               10.0.0               hc9558a2_0    conda-forge
llvmlite                  0.32.1                   pypi_0    pypi
matplotlib                3.2.1                    pypi_0    pypi
mkl                       2020.1                      217    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
networkx                  2.4                      pypi_0    pypi
ninja                     1.10.0               hc9558a2_0    conda-forge
numba                     0.49.1                   pypi_0    pypi
numpy                     1.18.4           py37h8960a57_0    conda-forge
openssl                   1.1.1g               h7b6447c_0
pcdet                     0.1.0+2244be4             dev_0    <develop>
pillow                    7.1.2                    pypi_0    pypi
pip                       20.1               pyh9f0ad1d_0    conda-forge
protobuf                  3.11.3                   pypi_0    pypi
pyparsing                 2.4.7                    pypi_0    pypi
python                    3.7.5                h0371630_0
python-dateutil           2.8.1                    pypi_0    pypi
python_abi                3.7                     1_cp37m    conda-forge
pytorch                   1.4.0           py3.7_cuda10.1.243_cudnn7.6.3_0    pytorch
pywavelets                1.1.1                    pypi_0    pypi
pyyaml                    5.3.1                    pypi_0    pypi
readline                  7.0               hf8c457e_1001    conda-forge
rhash                     1.3.6             h14c3975_1001    conda-forge
scikit-image              0.17.2                   pypi_0    pypi
scipy                     1.4.1                    pypi_0    pypi
setuptools                46.3.0           py37hc8dfbb8_0    conda-forge
six                       1.14.0                   pypi_0    pypi
spconv                    1.1                      pypi_0    pypi
sqlite                    3.31.1               h7b6447c_0
tensorboardx              2.0                      pypi_0    pypi
tifffile                  2020.5.11                pypi_0    pypi
tk                        8.6.10               hed695b0_0    conda-forge
tqdm                      4.46.0                   pypi_0    pypi
wheel                     0.34.2                     py_1    conda-forge
xz                        5.2.5                h516909a_0    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge

[MartinHahner]

Finally SOLVED!

The final issue was, that we tried to build spconv on a Tesla K40 GPU, which has a Kepler architecture and is seemingly too old to build spconv.

The reason why we always tried to build spconv on a Tesla K40 is that in our lab, it is hard to get a GPU on our GPU cluster interactively. (Usually, you can only submit jobs via qsub/SLURM.) So we always tried to build spconv either on our local Linux clients or on old nodes which only have Tesla K40s. Then, out of despair, we tried to build spconv on a Titan X (Pascal) and we finally found a combination of requirements that worked.

So here is our solution:

conda create --name PCDet python=3.6 pytorch=1.1 cudatoolkit=9.2 cudatoolkit-dev=9.2 \
cmake --channel pytorch --channel=conda-forge
conda activate PCDet
conda install cudnn
conda install boost
git clone https://github.com/traveller59/spconv spconv_8da6f96 --recursive
cd spconv_8da6f96
git checkout 8da6f967fb9a054d8870c3515b1b44eca2103634

If necessary: Download and extract 0001-Allow-to-specifiy-CUDA_ROOT-directory-and-pick-corre.patch.zip and patch spconv via: git am <PATH_TO_EXTRACTED_FILE>/0001-Allow-to-specifiy-CUDA_ROOT-directory-and-pick-corre.patch

CUDA_ROOT=<PATH_TO_YOUR_CONDA_INSTALLATION>/conda_envs/PCDet python setup.py bdist_wheel
cd dist/
pip install *

Test spconv via: python -c 'import spconv' (should just return and not raise any errors)

cd ../..
git clone https://github.com/sshaoshuai/PCDet.git
cd PCDet/
pip install -r requirements.txt
CUDA_ROOT=<PATH_TO_YOUR_CONDA_INSTALLATION>/conda_envs/PCDet python setup.py develop

Done!

I hope these instructions help someone else who struggles to build spconv as well.

[fregu856]

I also had trouble installing spconv 1.0, but switching to pytorch 1.1.0 solved that issue for me.

@fregu856: Which version of gcc were you using to build spconv? (gcc --version) Was it version 5.4 or another one?

gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609