COCO AP of FPN with ResNet-50 backbone for object detection

Hi @fmassa, thanks for the great codes. I am confused about COCO AP of Faster R-CNN ResNet-50 FPN, from Document and #925 and Source Code, I guess that the model Faster R-CNN ResNet-50 FPN was trained with following hyperparameters and got AP 37.0, am I right?

Repo	Network	box AP	scheduler	epochs	lr-steps	batch size	lr
vision	R-50 FPN	37.0	2x	26	16, 22	16	0.02

batch_size = 2 * 8 (NUM_GPU) = 16

However, I noticed that the box AP in maskrcnn-benchmark and Detectron seems to have better performance as below:

Repo	Network	box AP	scheduler	epochs	lr-steps	batch size	lr
maskrcnn-benchmark	R-50 FPN	36.8	1x	12.28	8.19, 10.92	16	0.02
Detectron	R-50 FPN	36.7	1x	12.28	8.19, 10.92	16	0.02
Detectron	R-50 FPN	37.9	2x	24.56	16.37, 21.83	16	0.02

from maskrcnn-benchmark 1x config epochs = 90000 (steps) * 16 (batch size) / 117266 (training images per epoch) = 12.28 btw, COCO2017 has 118287 training images but only 117266 training images contain at least one object

I would like to know what causes this gap?

• 37.0 (torchvision 2x) vs 36.8 (maskrcnn-benchmark 1x)
• 37.0 (torchvision 2x) vs 37.9 (Detectron 2x)

Besides, could I have the result which trained with scheduler 1x?

Repo	Network	box AP	scheduler	epochs	lr-steps	batch size	lr
vision	R-50 FPN	??	1x	13	8, 11	16	0.02

Thank you!

[potterhsu]

Sure, I’ve sent PR #2113 for this.

[fmassa]

Hi,

There are a few differences between both implementations that lead to this difference in mAP:

• maskrcnn-benchmark and detectron2 uses ResNet50 Caffe2 pre-trained weights with stride in the 1x1 convolution, while torchvision models uses the torchvision resnet50 models (with stride in the 3x3 convolution)
• we use l1 loss for both regression, instead of smooth_l1 – this is for simplicity, but gives a different trade-off on mAP and mAP@50
• we use slightly different weight initializations for some layers – this is for simplicity

Those all cumulate to lead to this discrepancy that you see. Given the complexity of Faster R-CNN as a model, every tiny detail can change a bit the dynamics of the training, while producing in the end (after more epochs) comparable models, so for the sake of uniformity and simplicity, we decided to make this compromise.

IIRC, training on the 1x schedule gives ~36.3 mAP, but I can’t find the logs anymore and would need to re-train the model to be sure.

Let me know if you have more questions!