Error in AMD GPU 6800xt(gfx1030) Rocm5.2.1 using mmcv 2.0.0rc1

Checklist

1. I have searched related issues #1394 but cannot get the expected help.
2. I have read the FAQ documentation but cannot get the expected help.
3. The unexpected results still exist in the latest version: mmcv 2.0.0.rc1

Describe the Issue

I created a new environment when I configured MMyolo, and after configuring it according to the documentation, when I run the demo program, the following error is reported if I specify the GPU as Cuda.

1 What command, code, or script did you run?

python demo/image_demo.py demo/demo.jpg \
                          yolov5_s-v61_syncbn_fast_8xb16-300e_coco.py \
                          yolov5_s-v61_syncbn_fast_8xb16-300e_coco_20220918_084700-86e02187.pth \
                          --device cuda \
                          --out-file result.jpg

1. Did you make any modifications on the code? Did you understand what you have modified? Just change ‘‘cpu’’ to “cuda”

Environment

1. my environment is below

sys.platform: linux
Python: 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0]
CUDA available: True
numpy_random_seed: 2147483648
GPU 0: AMD Radeon RX 6800 XT
CUDA_HOME: /opt/rocm-5.2.1
NVCC: Not Available
GCC: x86_64-linux-gnu-gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0
PyTorch: 1.12.1+rocm5.1.1
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - HIP Runtime 5.1.20531
  - MIOpen 2.16.0
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.1, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, 

TorchVision: 0.13.1+rocm5.1.1
OpenCV: 4.6.0
MMEngine: 0.1.0
MMCV: 2.0.0rc1
MMDetection: 3.0.0rc1
MMYOLO: 0.1.1+

1. You may add addition that may be helpful for locating the problem, such as I installed pytorch using pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/rocm5.1.1, and I tested it in yolov5 and yolov7 source code; both can normally run, including training and detecting.

Error traceback

Traceback (most recent call last):
  File "/media/zr/Data/MMLAB_2.0/mmyolo/demo/image_demo.py", line 61, in <module>
    main(args)
  File "/media/zr/Data/MMLAB_2.0/mmyolo/demo/image_demo.py", line 43, in main
    result = inference_detector(model, args.img)
  File "/media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/mmdet/apis/inference.py", line 152, in inference_detector
    results = model.test_step(data_)[0]
  File "/media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/mmengine/model/base_model/base_model.py", line 145, in test_step
    return self._run_forward(data, mode='predict')  # type: ignore
  File "/media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/mmengine/model/base_model/base_model.py", line 298, in _run_forward
    results = self(**data, mode=mode)
  File "/media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/mmdet/models/detectors/base.py", line 94, in forward
    return self.predict(inputs, data_samples)
  File "/media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/mmdet/models/detectors/single_stage.py", line 110, in predict
    results_list = self.bbox_head.predict(
  File "/media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/mmdet/models/dense_heads/base_dense_head.py", line 196, in predict
    predictions = self.predict_by_feat(
  File "/media/zr/Data/MMLAB_2.0/mmyolo/mmyolo/models/dense_heads/yolov5_head.py", line 406, in predict_by_feat
    results = self._bbox_post_process(
  File "/media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/mmdet/models/dense_heads/base_dense_head.py", line 478, in _bbox_post_process
    det_bboxes, keep_idxs = batched_nms(bboxes, results.scores,
  File "/media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/mmcv/ops/nms.py", line 334, in batched_nms
    dets, keep = nms_op(boxes_for_nms, scores, **nms_cfg_)
  File "/media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/mmengine/utils/misc.py", line 351, in new_func
    output = old_func(*args, **kwargs)
  File "/media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/mmcv/ops/nms.py", line 159, in nms
    inds = NMSop.apply(boxes, scores, iou_threshold, offset, score_threshold,
  File "/media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/mmcv/ops/nms.py", line 27, in forward
    inds = ext_module.nms(
RuntimeError: nms_impl: implementation for device cuda:0 not found.

Exception raised from Dispatch at /tmp/mmcv/mmcv/ops/csrc/common/pytorch_device_registry.hpp:122 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7fbae3043ab2 in /media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x5b (0x7fbae304014b in /media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: nms_impl(at::Tensor, at::Tensor, float, int) + 0xa97 (0x7fb9b64e2847 in /media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/mmcv/_ext.cpython-310-x86_64-linux-gnu.so)
frame #3: nms(at::Tensor, at::Tensor, float, int) + 0x4f (0x7fb9b64e2fcf in /media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/mmcv/_ext.cpython-310-x86_64-linux-gnu.so)
frame #4: <unknown function> + 0x12515b (0x7fb9b652515b in /media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/mmcv/_ext.cpython-310-x86_64-linux-gnu.so)
frame #5: <unknown function> + 0x11224f (0x7fb9b651224f in /media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/mmcv/_ext.cpython-310-x86_64-linux-gnu.so)
<omitting python frames>
frame #10: THPFunction_apply(_object*, _object*) + 0xb57 (0x7fbb5ef24937 in /media/zr/Data/MMLAB_2.0/venv/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #57: <unknown function> + 0x29d90 (0x7fbb82629d90 in /lib/x86_64-linux-gnu/libc.so.6)
frame #58: __libc_start_main + 0x80 (0x7fbb82629e40 in /lib/x86_64-linux-gnu/libc.so.6)

Now my device can only use CPU training and verification, and I want to know what I should do.

[zRzRzRzRzRzRzR]

My guess is that it might be because my torch is a package downloaded from the official website using pip, and I’ll try next to see if compiling the torch using the source code will fix the problem. Thank you for your kind help.

[zRzRzRzRzRzRzR]

Maybe it’s the python version or something, I was able to compile pytorch in my local environment and it works fine.This still confuses me.

~ pip list | grep torch            
torch                          1.12.1+rocm5.1.1
torchaudio                     0.12.1+rocm5.1.1
torchvision                    0.13.1+rocm5.1.1

~ python 
Python 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.get_device_properties(torch.device('cuda:0'))
_CudaDeviceProperties(name='AMD Radeon RX 6800 XT', major=10, minor=3, total_memory=16368MB, multi_processor_count=36)
>>> torch.cuda.is_available()
True