Low mAP on pytorch 1.0 branch

I downloaded the trained model from here.

model	#GPUs	batch size	lr	lr_decay	max_epoch	time/epoch	mem/GPU	mAP
VGG-16	1	1	1e-3	5	6	0.76 hr	3265MB	70.1

I use pytorch 1.0, and the mAP on pascal voc 2007 is 65.8, which is lower than 70.1. Here is the APs for each category.

VOC07 metric? Yes
AP for aeroplane = 0.6559
AP for bicycle = 0.7750
AP for bird = 0.6105
AP for boat = 0.4600
AP for bottle = 0.4542
AP for bus = 0.7689
AP for car = 0.7686
AP for cat = 0.8393
AP for chair = 0.4700
AP for cow = 0.6777
AP for diningtable = 0.6060
AP for dog = 0.7668
AP for horse = 0.8110
AP for motorbike = 0.6922
AP for person = 0.7295
AP for pottedplant = 0.3714
AP for sheep = 0.6060
AP for sofa = 0.6509
AP for train = 0.7408
AP for tvmonitor = 0.7092
Mean AP = 0.6582

I also tested on the master branch using pytorch 0.4.0. The results are as follows:

VOC07 metric? Yes
AP for aeroplane = 0.7055
AP for bicycle = 0.7761
AP for bird = 0.6638
AP for boat = 0.5457
AP for bottle = 0.5221
AP for bus = 0.8086
AP for car = 0.8450
AP for cat = 0.8411
AP for chair = 0.5025
AP for cow = 0.7824
AP for diningtable = 0.6539
AP for dog = 0.7754
AP for horse = 0.8294
AP for motorbike = 0.7267
AP for person = 0.7726
AP for pottedplant = 0.4223
AP for sheep = 0.7188
AP for sofa = 0.6555
AP for train = 0.7526
AP for tvmonitor = 0.7277
Mean AP = 0.7014

Is there something wrong about the evaluation code?

[adityaarun1]

Hi,

For the PyTorch-1.0 branch, I re-ran the code and it gives me 70.26% mAP when using VGG-16 and 74.90% using ResNet-101. These numbers are achieved using the default parameter values on a single GPU. These numbers are consistent with the ones reported by @jwyang in the readme. Here are the details:

VGG-16 (7th epoch)

VOC07 metric? Yes
AP for aeroplane = 0.7071
AP for bicycle = 0.7714
AP for bird = 0.6922
AP for boat = 0.5591
AP for bottle = 0.5510
AP for bus = 0.7913
AP for car = 0.8458
AP for cat = 0.8173
AP for chair = 0.4943
AP for cow = 0.7680
AP for diningtable = 0.6090
AP for dog = 0.8058
AP for horse = 0.8283
AP for motorbike = 0.7468
AP for person = 0.7788
AP for pottedplant = 0.4439
AP for sheep = 0.7058
AP for sofa = 0.6440
AP for train = 0.7584
AP for tvmonitor = 0.7345
Mean AP = 0.7026

ResNet-101 (7th epoch)

VOC07 metric? Yes
AP for aeroplane = 0.7883
AP for bicycle = 0.8002
AP for bird = 0.7752
AP for boat = 0.6471
AP for bottle = 0.6047
AP for bus = 0.8135
AP for car = 0.8585
AP for cat = 0.8698
AP for chair = 0.5443
AP for cow = 0.8122
AP for diningtable = 0.6728
AP for dog = 0.8568
AP for horse = 0.8484
AP for motorbike = 0.7912
AP for person = 0.7880
AP for pottedplant = 0.4702
AP for sheep = 0.7661
AP for sofa = 0.7292
AP for train = 0.7955
AP for tvmonitor = 0.7487
Mean AP = 0.7490

[DHZS]

Thank you very much, @adityaarun1. I saw you reply at https://github.com/ruotianluo/pytorch-faster-rcnn/pull/122. I trained a model from scratch and got 70.26% mAP. It seems that it is important to retrain the model in pytorch 1.0. Thanks again, you have helped me a lot.