_ext undefined symbol when using source-compiled pytorch

compiled pytorch v1.11.0 from source using CUDA 11.3, ubuntu 20.04, python 3.8 , installed to /usr/local/lib/python3.8/dist-packages/torch/

> nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_May__3_19:15:13_PDT_2021
Cuda compilation tools, release 11.3, V11.3.109
Build cuda_11.3.r11.3/compiler.29920130_0

I also installed torchvision from source (v0.12.0)

installed mmcv-full from pip wheels pip install mmcv-full==1.7.0 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.11/index.html

error whem running mmcv:

> python3 -c "from mmcv.utils import collect_env; print(collect_env())"
        Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/local/lib/python3.8/dist-packages/mmcv/utils/env.py", line 124, in collect_env
        from mmcv.ops import get_compiler_version, get_compiling_cuda_version
      File "/usr/local/lib/python3.8/dist-packages/mmcv/ops/__init__.py", line 2, in <module>
        from .active_rotated_filter import active_rotated_filter
      File "/usr/local/lib/python3.8/dist-packages/mmcv/ops/active_rotated_filter.py", line 10, in <module>
        ext_module = ext_loader.load_ext(
      File "/usr/local/lib/python3.8/dist-packages/mmcv/utils/ext_loader.py", line 13, in load_ext
        ext = importlib.import_module('mmcv.' + name)
      File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
    ImportError: /usr/local/lib/python3.8/dist-packages/mmcv/_ext.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN3c107Warning4warnERKNS_14SourceLocationERKSsb

> ldd /usr/local/lib/python3.8/dist-packages/mmcv/_ext.cpython-38-x86_64-linux-gnu.so 
        linux-vdso.so.1 (0x00007ffdf4141000)
        libc10.so => /usr/local/lib/python3.8/dist-packages/torch/lib/libc10.so (0x00007f0bb80ac000)
        libtorch.so => /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch.so (0x00007f0bb80a7000)
        libtorch_cpu.so => /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_cpu.so (0x00007f0bae5d7000)
        libtorch_python.so => /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_python.so (0x00007f0bad859000)
        libcudart.so.11.0 => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.11.0 (0x00007f0bad5ae000)
        libc10_cuda.so => /usr/local/lib/python3.8/dist-packages/torch/lib/libc10_cuda.so (0x00007f0bad556000)
        libtorch_cuda_cu.so => /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_cuda_cu.so (0x00007f0b7a57a000)
        libtorch_cuda_cpp.so => /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_cuda_cpp.so (0x00007f0b6c657000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f0b6c475000)
        libm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x00007f0b6c326000)
        libgcc_s.so.1 => /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0b6c30b000)
        libpthread.so.0 => /usr/lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0b6c2e6000)
        libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x00007f0b6c0f4000)
        libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f0b6c0e7000)
        libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f0b6c0a5000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f0bbd112000)
        libtorch_cuda.so => /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_cuda.so (0x00007f0b6c0a0000)
        libmpi_cxx.so.40 => /usr/lib/x86_64-linux-gnu/libmpi_cxx.so.40 (0x00007f0b6c082000)
        libmpi.so.40 => /usr/lib/x86_64-linux-gnu/libmpi.so.40 (0x00007f0b6bf5b000)
        librt.so.1 => /usr/lib/x86_64-linux-gnu/librt.so.1 (0x00007f0b6bf51000)
        libdl.so.2 => /usr/lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0b6bf4b000)
        libshm.so => /usr/local/lib/python3.8/dist-packages/torch/lib/libshm.so (0x00007f0b6bf41000)
        libnvToolsExt.so.1 => /usr/local/cuda/lib64/libnvToolsExt.so.1 (0x00007f0b6bd38000)
        libcudnn.so.8 => /usr/lib/x86_64-linux-gnu/libcudnn.so.8 (0x00007f0b6bb10000)
        libcusparse.so.11 => /usr/local/cuda/lib64/libcusparse.so.11 (0x00007f0b5df99000)
        libcurand.so.10 => /usr/local/cuda/lib64/libcurand.so.10 (0x00007f0b5880b000)
        libcusolver.so.11 => /usr/local/cuda/lib64/libcusolver.so.11 (0x00007f0b4b94f000)
        libnccl.so.2 => /usr/lib/x86_64-linux-gnu/libnccl.so.2 (0x00007f0b44291000)
        libcublas.so.11 => /usr/local/cuda/lib64/libcublas.so.11 (0x00007f0b3d24f000)
        libcufft.so.10 => /usr/local/cuda/lib64/libcufft.so.10 (0x00007f0b31a56000)
        libopen-rte.so.40 => /usr/lib/x86_64-linux-gnu/libopen-rte.so.40 (0x00007f0b3199c000)
        libopen-pal.so.40 => /usr/lib/x86_64-linux-gnu/libopen-pal.so.40 (0x00007f0b318ee000)
        libhwloc.so.15 => /usr/lib/x86_64-linux-gnu/libhwloc.so.15 (0x00007f0b3189d000)
        libcublasLt.so.11 => /usr/local/cuda/lib64/libcublasLt.so.11 (0x00007f0b25cbd000)
        libz.so.1 => /usr/lib/x86_64-linux-gnu/libz.so.1 (0x00007f0b25ca1000)
        libevent-2.1.so.7 => /usr/lib/x86_64-linux-gnu/libevent-2.1.so.7 (0x00007f0b25c4b000)
        libutil.so.1 => /usr/lib/x86_64-linux-gnu/libutil.so.1 (0x00007f0b25c46000)
        libevent_pthreads-2.1.so.7 => /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007f0b25c3f000)
        libudev.so.1 => /usr/lib/x86_64-linux-gnu/libudev.so.1 (0x00007f0b25c12000)
        libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007f0b25c07000)

[adizhol]

UPDATE solved problems by setting enviroment variable FORCE_CUDA=1
and TORCH_CUDA_ARCH_LIST=“3.5;5.0;6.0;6.1;7.0;7.5;8.0;8.6+PTX” (https://github.com/pytorch/extension-cpp/issues/71#issuecomment-1183674660)

Finally to build mmcv-full:

TORCH_CUDA_ARCH_LIST="3.5;5.0;6.0;6.1;7.0;7.5;8.0;8.6+PTX" \
MMCV_WITH_OPS=1 \
FORCE_CUDA=1 \
python setup.py install

[adizhol]

When I try to load/use a resnet+deformable convolution i get this error: modulated_deformable_im2col_impl: implementation for device cuda:1 not found

File "/root/miniconda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/miniconda/lib/python3.8/site-packages/mmcv_full-1.7.0-py3.8-linux-x86_64.egg/mmcv/ops/modulated_deform_conv.py", line 255, in forward
    return modulated_deform_conv2d(x, offset, mask, self.weight, self.bias,
  File "/root/miniconda/lib/python3.8/site-packages/mmcv_full-1.7.0-py3.8-linux-x86_64.egg/mmcv/ops/modulated_deform_conv.py", line 76, in forward
    ext_module.modulated_deform_conv_forward(
RuntimeError: modulated_deformable_im2col_impl: implementation for device cuda:1 not found.