Panoptic Segmentation Can Not be Exported to ONNX

Instructions To Reproduce the 🐛 Bug:

1. Full runnable code or full changes you made:

I use the original repo and do not change anything.

1. What exact command do you run:

export DETECTRON2_DATASETS=/data/datasets/

python3 export_model.py \
    --config-file ../../configs/COCO-PanopticSegmentation/panoptic_fpn_R_50_1x.yaml \
    --output /data/output/ \
    --export-method caffe2_tracing \
    --format onnx \
    MODEL.WEIGHTS /data/model/COCO-PanopticSegmentation/panoptic_fpn_R_50_1x/139514544/model_final_dbfeb4.pkl \
    MODEL.DEVICE cpu

I have already prepared the COCO dataset in /data/datasets/.

1. Full logs or other relevant observations:

[06/23 06:22:33 detectron2]: Command line arguments: Namespace(format='onnx', export_method='caffe2_tracing', config_file='../../configs/COCO-PanopticSegmentation/panoptic_fpn_R_50_1x.yaml', sample_image=None, run_eval=False, output='/data/output/', opts=['MODEL.WEIGHTS', '/data/model/COCO-PanopticSegmentation/panoptic_fpn_R_50_1x/139514544/model_final_dbfeb4.pkl', 'MODEL.DEVICE', 'cpu'])
[W init.cpp:759] Warning: Use _jit_set_fusion_strategy, bailout depth is deprecated. Setting to (STATIC, 1) (function operator())
[06/23 06:22:35 d2.data.datasets.coco]: Loaded 5000 images in COCO format from /data/datasets/coco/annotations/instances_val2017.json
[06/23 06:22:35 d2.data.datasets.coco]: Loaded 5000 images with semantic segmentation from /data/datasets/coco/val2017
[06/23 06:22:35 d2.data.build]: Distribution of instances among all 80 categories:
|   category    | #instances   |   category   | #instances   |   category    | #instances   |
|:-------------:|:-------------|:------------:|:-------------|:-------------:|:-------------|
|    person     | 10777        |   bicycle    | 314          |      car      | 1918         |
|  motorcycle   | 367          |   airplane   | 143          |      bus      | 283          |
|     train     | 190          |    truck     | 414          |     boat      | 424          |
| traffic light | 634          | fire hydrant | 101          |   stop sign   | 75           |
| parking meter | 60           |    bench     | 411          |     bird      | 427          |
|      cat      | 202          |     dog      | 218          |     horse     | 272          |
|     sheep     | 354          |     cow      | 372          |   elephant    | 252          |
|     bear      | 71           |    zebra     | 266          |    giraffe    | 232          |
|   backpack    | 371          |   umbrella   | 407          |    handbag    | 540          |
|      tie      | 252          |   suitcase   | 299          |    frisbee    | 115          |
|     skis      | 241          |  snowboard   | 69           |  sports ball  | 260          |
|     kite      | 327          | baseball bat | 145          | baseball gl.. | 148          |
|  skateboard   | 179          |  surfboard   | 267          | tennis racket | 225          |
|    bottle     | 1013         |  wine glass  | 341          |      cup      | 895          |
|     fork      | 215          |    knife     | 325          |     spoon     | 253          |
|     bowl      | 623          |    banana    | 370          |     apple     | 236          |
|   sandwich    | 177          |    orange    | 285          |   broccoli    | 312          |
|    carrot     | 365          |   hot dog    | 125          |     pizza     | 284          |
|     donut     | 328          |     cake     | 310          |     chair     | 1771         |
|     couch     | 261          | potted plant | 342          |      bed      | 163          |
| dining table  | 695          |    toilet    | 179          |      tv       | 288          |
|    laptop     | 231          |    mouse     | 106          |    remote     | 283          |
|   keyboard    | 153          |  cell phone  | 262          |   microwave   | 55           |
|     oven      | 143          |   toaster    | 9            |     sink      | 225          |
| refrigerator  | 126          |     book     | 1129         |     clock     | 267          |
|     vase      | 274          |   scissors   | 36           |  teddy bear   | 190          |
|  hair drier   | 11           |  toothbrush  | 57           |               |              |
|     total     | 36335        |              |              |               |              |
[06/23 06:22:35 d2.data.dataset_mapper]: [DatasetMapper] Augmentations used in inference: [ResizeShortestEdge(short_edge_length=(800, 800), max_size=1333, sample_style='choice')]
[06/23 06:22:35 d2.data.common]: Serializing 5000 elements to byte tensors and concatenating them all ...
[06/23 06:22:35 d2.data.common]: Serialized dataset takes 19.52 MiB
Traceback (most recent call last):
  File "/data/detectron2/tools/deploy/export_model.py", line 217, in <module>
    exported_model = export_caffe2_tracing(cfg, torch_model, sample_inputs)
  File "/data/detectron2/tools/deploy/export_model.py", line 36, in export_caffe2_tracing
    from detectron2.export import Caffe2Tracer
ImportError: cannot import name 'Caffe2Tracer' from 'detectron2.export' (/usr/local/lib/python3.10/dist-packages/detectron2-0.6-py3.10-linux-x86_64.egg/detectron2/export/__init__.py)

1. please simplify the steps as much as possible so they do not require additional resources to run, such as a private dataset.

Expected behavior:

I expected that the official code export_model.py can help me to export onnx model.

Environment:

Provide your environment information using the following command:

---------------------  ----------------------------------------------------------------------------------------------
sys.platform           linux
Python                 3.10.4 (main, Apr  2 2022, 09:04:19) [GCC 11.2.0]
numpy                  1.22.4
detectron2             0.6 @/usr/local/lib/python3.10/dist-packages/detectron2-0.6-py3.10-linux-x86_64.egg/detectron2
Compiler               GCC 11.2
CUDA compiler          not available
DETECTRON2_ENV_MODULE  <not set>
PyTorch                1.11.0+cpu @/usr/local/lib/python3.10/dist-packages/torch
PyTorch debug build    False
GPU available          No: torch.cuda.is_available() == False
Pillow                 9.1.1
torchvision            0.12.0+cpu @/usr/local/lib/python3.10/dist-packages/torchvision
fvcore                 0.1.5.post20220512
iopath                 0.1.9
cv2                    4.6.0
---------------------  ----------------------------------------------------------------------------------------------
PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.5.2 (Git Hash a9302535553c73243c632ad3c4c80beec3d19a1e)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.11.0, USE_CUDA=0, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 

If your issue looks like an installation issue / environment issue, please first try to solve it yourself with the instructions in https://detectron2.readthedocs.io/tutorials/install.html#common-installation-issues

Finally, thanks to your effects that contribute such excellent repo!

[TankAbraham]

I will not leave and keep the focus on until this problem has been solved.

[thiagocrepaldi]

@thiagocrepaldi thanks I managed to export the PanopticFPN using your branch. Regarding the If operator following the first NMS in the onnx graph:

It points to this line https://github.com/pytorch/vision/blob/07ae61bf9c21ddd1d5f65d326aa9636849b383ca/torchvision/ops/boxes.py#L89

After disabling this line in the source code (for debugging purposes), the If operator is still there but pointing to another line:

https://github.com/pytorch/vision/blob/07ae61bf9c21ddd1d5f65d326aa9636849b383ca/torchvision/ops/boxes.py#L41

Curiously, when I export the nms in a standalone network (so simple pure pytorch network), there is no If operator in the onnx graph. Do you have an idea where this originates from?

In the network, the If op. always (10 times) occurs in the same subgraph so I wonder if this subgraph gets expanded from a single LOC in pytorch/python?

I am glad the PR worked out for you, but I am not sure if I undertood what you meant by “After disabling this line in the source code (for debugging purposes), the If operator is still there but pointing to another line”. If you comment it out this if, you change the graph and not necessarily If_1610 in one run matches the other. The numbering is kind of sequential counter, not actually connected to the source-code