Inference not correct on exported onnx model

Instructions To Reproduce the Issue:

To reproduce this behaviour please also refer to the following notebook: https://github.com/randombenj/detectron2onnx-inference/blob/master/detectron2-onnx.ipynb

When using the Caffe2Tracer to export to an onnx trace model, the model does not perform as expected (with the default trained weights). The inference behaviour is different than when running the inference with the DefaultPredictor.

Configure model and weights:

model_config = "COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml"

cfg = get_cfg()

# add project-specific config (e.g., TensorMask) here if you're not running a model in detectron2's core library
cfg.merge_from_file(model_zoo.get_config_file(model_config))
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5  # set threshold for this model

# Find a model from detectron2's model zoo. You can use the https://dl.fbaipublicfiles... url as well
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(model_config)
cfg.MODEL.DEVICE = 'cpu'

Load and prepare image:

im = cv2.imread('datasets/coco/val2017/000000023937.jpg')
batch_size, channels, height, width = 1, 3, 480, 640

im = cv2.resize(im, (width, height))
image = torch.as_tensor(im.astype("float32")).view(batch_size, height, width, channels)

# NWHC -> NCHW
image = image.permute(0, 3, 1, 2)
# img_info for inference
image_info = torch.tensor(
    [width, height, 1.0]
).to(dtype=torch.float32).view(batch_size, 3)

Run inference using DefaultPredictor:

predictor = DefaultPredictor(cfg)
outputs = predictor(im)

Export to onnx:

torch_model = build_model(cfg)
DetectionCheckpointer(torch_model).resume_or_load(cfg.MODEL.WEIGHTS)
torch_model.eval()

tracer = Caffe2Tracer(
    cfg, 
    torch_model, 
    [{"image": image[0]}]
)

Run onnx inference:

model = onnx.load("model.onnx")  # or use the `onnx_model` from above
onnx.checker.check_model(model)

W = {
    model.graph.input[0].name: image.data.numpy(),
    model.graph.input[1].name: image_info.data.numpy()
}

rep = backend.prepare(model)
onnx_outputs = rep.run(W)

• Am I doing something wrong with the tracing (input samples), or in the onnx inference?
• Could this behaviour be due to tracing and not scripting export?

Expected behavior:

I would not expect such a big difference in inference accuracy, otherwise, the export is kind of pointless.

Environment:

Paste the output of the following command:

---------------------  ---------------------------------------------------------
sys.platform           linux
Python                 3.6.9 (default, Jan 26 2021, 15:33:00) [GCC 8.4.0]
numpy                  1.19.2
detectron2             0.3 @/home/users/benj/Work/detectron2/detectron2
Compiler               GCC 7.5
CUDA compiler          not available
DETECTRON2_ENV_MODULE  <not set>
PyTorch                1.8.0 @/usr/local/lib/python3.6/dist-packages/torch
PyTorch debug build    False
GPU available          False
Pillow                 7.2.0
torchvision            0.9.0 @/usr/local/lib/python3.6/dist-packages/torchvision
fvcore                 0.1.3.post20210310
cv2                    4.4.0
---------------------  ---------------------------------------------------------
PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

[randombenj]

Thanks for the hint with the resizing, turns out this was actually what caused the difference in accuracy. For anyone stumbling upon this issue here is how you can run caffe2 and onnx inference:

# get a sample data
data_loader = build_detection_test_loader(cfg, cfg.DATASETS.TEST[0])
first_batch = next(iter(data_loader))

# trace both caffe2 and onnx
tracer = Caffe2Tracer(
    cfg, 
    torch_model, 
    first_batch
)

caffe2_model = tracer.export_caffe2()
onnx_model = tracer.export_onnx()

# now for the tricky part (the input) 
original_image = cv2.imread('datasets/coco/val2017/000000023937.jpg')

aug = T.ResizeShortestEdge(
    [cfg.INPUT.MIN_SIZE_TEST, cfg.INPUT.MIN_SIZE_TEST], cfg.INPUT.MAX_SIZE_TEST
)

height, width = original_image.shape[:2]
image = aug.get_transform(original_image).apply_image(original_image)
image = torch.as_tensor(image.astype("float32").transpose(2, 0, 1))

inputs = {"image": image, "height": height, "width": width}

# onnx need this extra step (that's) what I was missing
(image, img_info) = convert_batched_inputs_to_c2_format([inputs], 32, "cpu")

# run caffe2 inference
caffe2_predictions = caffe2_model([inputs])[0]

# run onnx inference
W = {
    onnx_model.graph.input[0].name: image.numpy(),
    onnx_model.graph.input[1].name: img_info.numpy()
}
rep = backend.prepare(onnx_model)
onnx_outputs = rep.run(W)

Now I get the exact same inference as with the DefaultPredictor

[Chao86]

I dont think onnxruntime is supported and you have to use the caffe2 onnx backend for running inference with detectron2 models.

hello, I’m using backend.prepare to run inference, however, there is a error in the termial.

Code:

Terminal As following:

I have no idea how that came about , can you offer some help?