[Bug] mmdetection DCNv2 convert to TensorRT using mmdeploy converter resulting lower bbox AP

Checklist

• I have searched related issues but cannot get the expected help.
• 2. I have read the FAQ documentation but cannot get the expected help.
• 3. The bug has not been fixed in the latest version.

Describe the bug

mmdetection DCNv2 convert to TensorRT using mmdeploy converter resulting lower bbox AP. The bbox AP of PyTorch model on coco val is 41.5, but the TensorRT model has only 34.6.

Reproduction

For PyTorch inference

python tools/test.py \
    configs/dcnv2/faster_rcnn_r50_fpn_mdconv_c3-c5_group4_1x_coco.py \
    work_dirs/dcnv2/latest.pth \
    --eval bbox

For TensorRT converting

python ../mmdeploy/tools/deploy.py \
    ../mmdeploy/configs/mmdet/detection/detection_tensorrt-fp16_dynamic-320x320-1344x1344.py \
    configs/dcnv2/faster_rcnn_r50_fpn_mdconv_c3-c5_group4_1x_coco.py \
    work_dirs/dcnv2/latest.pth \
    demo/demo.jpg \
    --work-dir ../mmdeploy_model/test/tensorrt/fp16 \
    --device cuda:0 \
    --dump-info

For TensorRT inference

python ../mmdeploy/tools/test.py \
    ../mmdeploy/configs/mmdet/detection/detection_tensorrt-fp16_dynamic-320x320-1344x1344.py \
    configs/dcnv2/faster_rcnn_r50_fpn_mdconv_c3-c5_group4_1x_coco.py \
    --model ../mmdeploy_model/test/tensorrt/fp16/end2end.engine \
    --metrics bbox \
    --device cuda \
    --speed-test

Environment

2022-12-05 16:27:24,843 - mmdeploy - INFO -

2022-12-05 16:27:24,843 - mmdeploy - INFO - **********Environmental information**********
2022-12-05 16:27:52,622 - mmdeploy - INFO - sys.platform: linux
2022-12-05 16:27:52,622 - mmdeploy - INFO - Python: 3.8.13 (default, Mar 28 2022, 11:38:47) [GCC 7.5.0]
2022-12-05 16:27:52,622 - mmdeploy - INFO - CUDA available: True
2022-12-05 16:27:52,622 - mmdeploy - INFO - GPU 0: GRID T4-8C
2022-12-05 16:27:52,623 - mmdeploy - INFO - CUDA_HOME: /usr/local/cuda-11.0
2022-12-05 16:27:52,623 - mmdeploy - INFO - NVCC: Cuda compilation tools, release 11.0, V11.0.194
2022-12-05 16:27:52,623 - mmdeploy - INFO - GCC: gcc (GCC) 7.5.0
2022-12-05 16:27:52,623 - mmdeploy - INFO - PyTorch: 1.12.1
2022-12-05 16:27:52,623 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.3
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
  - CuDNN 8.3.2  (built against CUDA 11.5)
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

2022-12-05 16:27:52,623 - mmdeploy - INFO - TorchVision: 0.13.1
2022-12-05 16:27:52,623 - mmdeploy - INFO - OpenCV: 4.6.0
2022-12-05 16:27:52,623 - mmdeploy - INFO - MMCV: 1.7.0
2022-12-05 16:27:52,623 - mmdeploy - INFO - MMCV Compiler: GCC 7.5
2022-12-05 16:27:52,623 - mmdeploy - INFO - MMCV CUDA Compiler: 11.0
2022-12-05 16:27:52,623 - mmdeploy - INFO - MMDeploy: 0.11.0+104f53e
2022-12-05 16:27:52,623 - mmdeploy - INFO -

2022-12-05 16:27:52,623 - mmdeploy - INFO - **********Backend information**********
2022-12-05 16:28:40,915 - mmdeploy - INFO - onnxruntime: 1.12.1 ops_is_avaliable : True
2022-12-05 16:28:41,772 - mmdeploy - INFO - tensorrt: 8.2.3.0   ops_is_avaliable : True
2022-12-05 16:28:42,108 - mmdeploy - INFO - ncnn: 1.0.20220909  ops_is_avaliable : False
2022-12-05 16:28:42,215 - mmdeploy - INFO - pplnn_is_avaliable: False
2022-12-05 16:28:42,496 - mmdeploy - INFO - openvino_is_avaliable: True
2022-12-05 16:28:42,858 - mmdeploy - INFO - snpe_is_available: False
2022-12-05 16:28:42,934 - mmdeploy - INFO - ascend_is_available: False
2022-12-05 16:28:42,983 - mmdeploy - INFO - coreml_is_available: False
2022-12-05 16:28:42,983 - mmdeploy - INFO -

2022-12-05 16:28:42,983 - mmdeploy - INFO - **********Codebase information**********
2022-12-05 16:28:43,030 - mmdeploy - INFO - mmdet:      2.26.0
2022-12-05 16:28:43,031 - mmdeploy - INFO - mmseg:      0.29.1
2022-12-05 16:28:43,031 - mmdeploy - INFO - mmcls:      None
2022-12-05 16:28:43,031 - mmdeploy - INFO - mmocr:      None
2022-12-05 16:28:43,031 - mmdeploy - INFO - mmedit:     None
2022-12-05 16:28:43,031 - mmdeploy - INFO - mmdet3d:    None
2022-12-05 16:28:43,031 - mmdeploy - INFO - mmpose:     None
2022-12-05 16:28:43,031 - mmdeploy - INFO - mmrotate:   None
2022-12-05 16:28:43,031 - mmdeploy - INFO - mmaction:   None

Error traceback

I acknowledged there were some related info during PyTorch inference.

2022-12-05 15:31:26,124 - root - INFO - ModulatedDeformConvPack backbone.layer2.0.conv2 is upgraded to version 2.

During TensorRT converting, there were also some related warnings.

xxx/code/mmcv/mmcv/ops/modulated_deform_conv.py:216: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version o
f pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, r
ounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  output_size += ((in_size + (2 * pad) - kernel) // stride_ + 1, )
xxx/code/mmcv/mmcv/ops/modulated_deform_conv.py:217: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrec
t. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if not all(map(lambda s: s > 0, output_size)):
...
[12/05/2022-15:42:23] [TRT] [I] No importer registered for op: MMCVModulatedDeformConv2d. Attempting to import as plugin.
[12/05/2022-15:42:23] [TRT] [I] Searching for plugin: MMCVModulatedDeformConv2d, plugin_version: 1, plugin_namespace:
[12/05/2022-15:42:23] [TRT] [I] Successfully created plugin: MMCVModulatedDeformConv2d

[vedrusss]

I’ve done that. Just needed to rescale detected boxes to original image shape. Now I can prove: transformed to TRT dyhead model works fine, boxes and scores are same as obtained from original model. There are some small discrepancies for some detections, I believe they are due to small numerical discrepancies appeared during model transformation and this is normal. Thanks to @hanrui1sensetime @grimoire

BTW, currently I’ve tested only FP32 mode. Gonna do same test for FP16.

[hanrui1sensetime]

#1493 can fix it. Thank to @grimoire