ImportError: libtorch_cuda_cu.so: cannot open shared object file

Hi everyone, I have successfully installed MMDetection3D, but could not run the demo code due to the error ImportError: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory. The details can be found below. I do appreciate if anyone could help me out. Thanks.

[UPDATE] You can find the solution at the end of this post.

Describe the bug When I ran demo/pcd_demo.py or mmdet3d/utils/collect_env.py, I got the error ImportError: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory.

Reproduction

1. How I installed MMDetection3D: Pull the PyTorch docker image with tag 20.12-py3 from NVIDIA NGC: docker pull nvcr.io/nvidia/pytorch:20.12-py3. Launch a docker container.

apt update
apt install ffmpeg libsm6 libxext6 git ninja-build libglib2.0-0 libsm6 libxrender-dev libxext6
apt clean
rm -rf /var/lib/apt/lists/*
pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.8.0/index.html
pip install mmdet
conda clean --all
git clone https://github.com/open-mmlab/mmdetection3d.git
cd mmdetection3d
pip install -r requirements/build.txt
pip install --no-cache-dir -v -e .
pip uninstall pycocotools --no-cache-dir
pip install mmpycocotools --no-cache-dir --force --no-deps

Create 2 directories named checkpoints and results inside mmdetection3d. Download a pretrained model and save it in checkpoints.

1. How I got the error: run either

python demo/pcd_demo.py demo/data/kitti/kitti_000008.bin configs/second/hv_second_secfpn_6x8_80e_kitti-3d-car.py checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-car_20200620_230238-393f000c.pth --out-dir results

or

python mmdet3d/utils/collect_env.py

Environment

1. I could not run python mmdet3d/utils/collect_env.py to collect necessary environment infomation as suggested.
2. OS: Ubuntu 20.04.2 LTS
3. GPU: RTX 3070. Driver: 460.32.03.
4. cuDNN 8.0.5; CUDA 11.1.1 including cuBLAS 11.3.0.
5. Python 3.8.5.
6. PyTorch 1.8.0a0+1606899.

Error traceback When running demo/pcd_demo.py:

Traceback (most recent call last):
  File "demo/pcd_demo.py", line 3, in <module>
    from mmdet3d.apis import inference_detector, init_detector, show_result_meshlab
  File "/mydev/code/mmdetection3d/mmdet3d/apis/__init__.py", line 1, in <module>
    from .inference import (convert_SyncBN, inference_detector,
  File "/mydev/code/mmdetection3d/mmdet3d/apis/inference.py", line 10, in <module>
    from mmdet3d.core import (Box3DMode, DepthInstance3DBoxes,
  File "/mydev/code/mmdetection3d/mmdet3d/core/__init__.py", line 1, in <module>
    from .anchor import *  # noqa: F401, F403
  File "/mydev/code/mmdetection3d/mmdet3d/core/anchor/__init__.py", line 1, in <module>
    from mmdet.core.anchor import build_anchor_generator
  File "/opt/conda/lib/python3.8/site-packages/mmdet/core/__init__.py", line 2, in <module>
    from .bbox import *  # noqa: F401, F403
  File "/opt/conda/lib/python3.8/site-packages/mmdet/core/bbox/__init__.py", line 7, in <module>
    from .samplers import (BaseSampler, CombinedSampler,
  File "/opt/conda/lib/python3.8/site-packages/mmdet/core/bbox/samplers/__init__.py", line 9, in <module>
    from .score_hlr_sampler import ScoreHLRSampler
  File "/opt/conda/lib/python3.8/site-packages/mmdet/core/bbox/samplers/score_hlr_sampler.py", line 2, in <module>
    from mmcv.ops import nms_match
  File "/opt/conda/lib/python3.8/site-packages/mmcv/ops/__init__.py", line 1, in <module>
    from .bbox import bbox_overlaps
  File "/opt/conda/lib/python3.8/site-packages/mmcv/ops/bbox.py", line 3, in <module>
    ext_module = ext_loader.load_ext('_ext', ['bbox_overlaps'])
  File "/opt/conda/lib/python3.8/site-packages/mmcv/utils/ext_loader.py", line 11, in load_ext
    ext = importlib.import_module('mmcv.' + name)
  File "/opt/conda/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory

When running mmdet3d/utils/collect_env.py:

Traceback (most recent call last):
  File "mmdet3d/utils/collect_env.py", line 18, in <module>
    for name, val in collect_env().items():
  File "mmdet3d/utils/collect_env.py", line 10, in collect_env
    env_info = collect_base_env()
  File "/opt/conda/lib/python3.8/site-packages/mmcv/utils/env.py", line 85, in collect_env
    from mmcv.ops import get_compiler_version, get_compiling_cuda_version
  File "/opt/conda/lib/python3.8/site-packages/mmcv/ops/__init__.py", line 1, in <module>
    from .bbox import bbox_overlaps
  File "/opt/conda/lib/python3.8/site-packages/mmcv/ops/bbox.py", line 3, in <module>
    ext_module = ext_loader.load_ext('_ext', ['bbox_overlaps'])
  File "/opt/conda/lib/python3.8/site-packages/mmcv/utils/ext_loader.py", line 11, in load_ext
    ext = importlib.import_module('mmcv.' + name)
  File "/opt/conda/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory

[UPDATE] Solution Following Tai-Wang’s suggestion, I re-built all 3 libraries mmcv, mmdet and mmdet3d from source and the aforementioned error was fixed. In summary, my step-by-step installation is as follows: Pull the PyTorch docker image with tag 20.12-py3 from NVIDIA NGC: docker pull nvcr.io/nvidia/pytorch:20.12-py3. Launch a docker container.

apt update
apt install ffmpeg libsm6 libxext6 git ninja-build libglib2.0-0 libsm6 libxrender-dev libxext6
apt clean
rm -rf /var/lib/apt/lists/*
git clone https://github.com/open-mmlab/mmcv.git
cd mmcv
MMCV_WITH_OPS=1 pip install -e .
cd ..
git clone https://github.com/open-mmlab/mmdetection.git
cd mmdetection
pip install -r requirements/build.txt
pip install -v -e .
git clone https://github.com/open-mmlab/mmdetection3d.git
cd mmdetection3d
pip install -v -e .

[Tai-Wang]

Quite interesting. You can successfully import mmcv while fail to do it in the mmdet3d (from the error message). It seems like there are still some problems with your installed mmcv. Maybe you can try to uninstall mmcv and mmdet and reinstall it by building from source, like:

pip uninstall mmcv-full
pip uninstall mmdet
git clone https://github.com/open-mmlab/mmcv.git
cd mmcv
MMCV_WITH_OPS=1 pip install -e .  # package mmcv-full will be installed after this step
cd ..
git clone https://github.com/open-mmlab/mmdetection.git
cd mmdetection
pip install -r requirements/build.txt
pip install -v -e .

Then build mmdet3d again. In this way, we can guarantee the compatibility of compilation environment with mmcv/mmdet/mmdet3d. BTW, you can also refer to the trouble shooting page of mmcv, which maybe useful for you.

[Wuziyi616]

I am trying to reproduce your error so please be patient. One thing I want to make sure, can you successfully import torch, mmcv and mmdet? Can you try running some simple demos of mmdet to make sure whether the question really stems from mmdet3d? Thanks!