CUDA error: no kernel image is available for execution on the device

Training PointPillars and SECOND works fine, but when I try to train PartA^2Net, I get this error message after addressing issue #70:

Traceback (most recent call last):
File "train.py", line 215, in <module>
main()
File "train.py", line 210, in main
max_ckpt_save_num=arguments.max_ckpt_save_num)
File "/scratch_net/hox/mhahner/repositories/PCDet/tools/train_utils/train_utils.py", line 80, in train_model
leave_pbar=(cur_epoch + 1 == total_epochs)
File "/scratch_net/hox/mhahner/repositories/PCDet/tools/train_utils/train_utils.py", line 36, in train_one_epoch
loss, tb_dict, disp_dict = model_func(model, batch)
File "/scratch_net/hox/mhahner/repositories/PCDet/pcdet/models/__init__.py", line 25, in model_func
ret_dict, tb_dict, disp_dict = model(input_dict)
File "/home/mhahner/scratch/apps/anaconda3/envs/spconv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(*input, **kwargs)
File "/scratch_net/hox/mhahner/repositories/PCDet/pcdet/models/detectors/PartA2_net.py", line 112, in forward
batch_size, voxel_centers, coords, rpn_ret_dict, input_dict
File "/scratch_net/hox/mhahner/repositories/PCDet/pcdet/models/detectors/PartA2_net.py", line 98, in forward_rcnn
rcnn_ret_dict = self.rcnn_net.forward(rcnn_input_dict)
File "/scratch_net/hox/mhahner/repositories/PCDet/pcdet/models/rcnn/partA2_rcnn_net.py", line 323, in forward
targets_dict = self.assign_targets(batch_size, rcnn_dict)
File "/scratch_net/hox/mhahner/repositories/PCDet/pcdet/models/rcnn/partA2_rcnn_net.py", line 27, in assign_targets
targets_dict = proposal_target_layer(rcnn_dict, roi_sampler_cfg=self.rcnn_target_config)
File "/scratch_net/hox/mhahner/repositories/PCDet/pcdet/models/model_utils/proposal_target_layer.py", line 14, in proposal_target_layer
sample_rois_for_rcnn(rois, gt_boxes, roi_raw_scores, roi_labels, roi_sampler_cfg)
File "/scratch_net/hox/mhahner/repositories/PCDet/pcdet/models/model_utils/proposal_target_layer.py", line 82, in sample_rois_for_rcnn
cur_gt[:, 0:7], cur_gt_labels)
File "/scratch_net/hox/mhahner/repositories/PCDet/pcdet/models/model_utils/proposal_target_layer.py", line 183, in get_maxiou3d_with_same_class
iou3d = iou3d_nms_utils.boxes_iou3d_gpu(cur_roi, cur_gt) # (M, N)
File "/scratch_net/hox/mhahner/repositories/PCDet/pcdet/ops/iou3d_nms/iou3d_nms_utils.py", line 47, in boxes_iou3d_gpu
overlaps_h = torch.clamp(min_of_max - max_of_min, min=0)
RuntimeError: CUDA error: no kernel image is available for execution on the device

My conda environment looks like this:

# packages in environment at /home/mhahner/scratch/apps/anaconda3/envs/spconv:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                      1_llvm    conda-forge
beautifulsoup4            4.9.1                    pypi_0    pypi
bzip2                     1.0.8                h516909a_2    conda-forge
ca-certificates           2020.1.1                      0
cachetools                4.1.0                    pypi_0    pypi
certifi                   2020.4.5.1               py37_0
chardet                   3.0.4                    pypi_0    pypi
cmake                     3.17.0               h28c56e5_0    conda-forge
coloredlogs               14.0                     pypi_0    pypi
cudatoolkit               10.1.243             h6bb024c_0
cudatoolkit-dev           10.1.243             h516909a_3    conda-forge
cudnn                     7.6.5                cuda10.1_0
cycler                    0.10.0                   pypi_0    pypi
decorator                 4.4.2                    pypi_0    pypi
easydict                  1.9                      pypi_0    pypi
expat                     2.2.9                he1b5a44_2    conda-forge
google                    2.0.3                    pypi_0    pypi
google-auth               1.15.0                   pypi_0    pypi
google-auth-oauthlib      0.4.1                    pypi_0    pypi
gspread                   3.6.0                    pypi_0    pypi
httplib2                  0.18.0                   pypi_0    pypi
humanfriendly             8.2                      pypi_0    pypi
idna                      2.9                      pypi_0    pypi
imagecodecs               2020.2.18                pypi_0    pypi
imageio                   2.8.0                    pypi_0    pypi
kiwisolver                1.2.0                    pypi_0    pypi
krb5                      1.17.1               h2fd8d38_0    conda-forge
ld_impl_linux-64          2.34                 h53a641e_0    conda-forge
libblas                   3.8.0               16_openblas    conda-forge
libcblas                  3.8.0               16_openblas    conda-forge
libcurl                   7.69.1               hf7181ac_0    conda-forge
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libffi                    3.2.1             he1b5a44_1007    conda-forge
libgcc-ng                 9.2.0                h24d8f2e_2    conda-forge
libgfortran-ng            7.5.0                hdf63c60_6    conda-forge
liblapack                 3.8.0               16_openblas    conda-forge
libopenblas               0.3.9                h5ec1e0e_0    conda-forge
libssh2                   1.9.0                hab1572f_2    conda-forge
libstdcxx-ng              9.2.0                hdf63c60_2    conda-forge
libuv                     1.34.0               h516909a_0    conda-forge
llvm-openmp               10.0.0               hc9558a2_0    conda-forge
llvmlite                  0.32.1                   pypi_0    pypi
matplotlib                3.2.1                    pypi_0    pypi
mkl                       2020.1                      219    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
networkx                  2.4                      pypi_0    pypi
ninja                     1.10.0               hc9558a2_0    conda-forge
numba                     0.49.1                   pypi_0    pypi
numpy                     1.18.4           py37h8960a57_0    conda-forge
oauth2client              4.1.3                    pypi_0    pypi
oauthlib                  3.1.0                    pypi_0    pypi
openssl                   1.1.1g               h7b6447c_0
pillow                    7.1.2                    pypi_0    pypi
pip                       20.1.1             pyh9f0ad1d_0    conda-forge
protobuf                  3.12.0                   pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pyparsing                 2.4.7                    pypi_0    pypi
python                    3.7.6           h8356626_5_cpython    conda-forge
python-dateutil           2.8.1                    pypi_0    pypi
python_abi                3.7                     1_cp37m    conda-forge
pytorch                   1.4.0           py3.7_cuda10.1.243_cudnn7.6.3_0    pytorch
pywavelets                1.1.1                    pypi_0    pypi
pyyaml                    5.3.1                    pypi_0    pypi
readline                  8.0                  hf8c457e_0    conda-forge
requests                  2.23.0                   pypi_0    pypi
requests-oauthlib         1.3.0                    pypi_0    pypi
rhash                     1.3.6             h14c3975_1001    conda-forge
rsa                       4.0                      pypi_0    pypi
scikit-image              0.17.2                   pypi_0    pypi
scipy                     1.4.1                    pypi_0    pypi
setuptools                46.4.0           py37hc8dfbb8_0    conda-forge
six                       1.14.0                   pypi_0    pypi
soupsieve                 2.0.1                    pypi_0    pypi
spconv                    1.0                      pypi_0    pypi
sqlite                    3.30.1               hcee41ef_0    conda-forge
tensorboardx              2.0                      pypi_0    pypi
tifffile                  2020.5.11                pypi_0    pypi
tk                        8.6.10               hed695b0_0    conda-forge
tqdm                      4.46.0                   pypi_0    pypi
urllib3                   1.25.9                   pypi_0    pypi
wheel                     0.34.2                     py_1    conda-forge
xz                        5.2.5                h516909a_0    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge

[MartinHahner]

Finally SOLVED!

The final issue was, that we tried to build spconv on a Tesla K40 GPU, which has a Kepler architecture and is seemingly too old to build spconv.

The reason why we always tried to build spconv on a Tesla K40 is that in our lab, it is hard to get a GPU on our GPU cluster interactively. (Usually, you can only submit jobs via qsub/SLURM.) So we always tried to build spconv either on our local Linux clients or on old nodes which only have Tesla K40s. Then, out of despair, we tried to build spconv on a Titan X (Pascal) and we finally found a combination of requirements that worked.

So here is our solution:

conda create --name PCDet python=3.6 pytorch=1.1 cudatoolkit=9.2 cudatoolkit-dev=9.2 \
cmake --channel pytorch --channel=conda-forge
conda activate PCDet
conda install cudnn
conda install boost
git clone https://github.com/traveller59/spconv spconv_8da6f96 --recursive
cd spconv_8da6f96
git checkout 8da6f967fb9a054d8870c3515b1b44eca2103634

If necessary: Download and extract 0001-Allow-to-specifiy-CUDA_ROOT-directory-and-pick-corre.patch.zip and patch spconv via: git am <PATH_TO_EXTRACTED_FILE>/0001-Allow-to-specifiy-CUDA_ROOT-directory-and-pick-corre.patch

CUDA_ROOT=<PATH_TO_YOUR_CONDA_INSTALLATION>/conda_envs/PCDet python setup.py bdist_wheel
cd dist/
pip install *

Test spconv via: python -c 'import spconv' (should just return and not raise any errors)

cd ../..
git clone https://github.com/sshaoshuai/PCDet.git
cd PCDet/
pip install -r requirements.txt
CUDA_ROOT=<PATH_TO_YOUR_CONDA_INSTALLATION>/conda_envs/PCDet python setup.py develop

Done!

I hope these instructions help someone else who struggles to build spconv as well.

[KangChou]