Problems training Faster-RCNN from pretrained backbone

Is there any recommendation to train Faster-RCNN starting from the pretrained backbone? I’m using VOC 2007 dataset and I’m able to do transfer learning starting from:

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = torchvision.models.detection.faster_rcnn.FastRCNNPredictor(in_features, num_classes=21)

Using COCO pretrained ‘fasterrcnn_resnet50_fpn’ i’m able to obtain an mAP of 79% on VOC 2007 test set. Problems arise when i try to train from scratch using only the pretrained backbone:

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = torchvision.models.detection.faster_rcnn.FastRCNNPredictor(in_features, num_classes=21)

I have been trying to train this model for weeks but the highest mAP i got was 63% (again on test set).

Now, i know that training from scratch is harder, but i really would like to know how to set the training parameters to obtain a decent accuracy, in the future i may want to change the backbone and chances are that i will be not able to find a pretrained faster-rcnn on which i can do transfer learning.

[lpuglia]

@hktxt FYI i can get easily 72% mAP using the example provided in FasterRCNN source code using mobilenet_v2 as backbone:

    backbone = torchvision.models.mobilenet_v2(pretrained=True).features
    backbone.out_channels = 1280
    anchor_generator = AnchorGenerator(sizes=((32, 64, 128, 256, 512),),
                                       aspect_ratios=((0.5, 1.0, 2.0),))
    roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                    output_size=7,
                                                    sampling_ratio=2)
    model = torchvision.models.detection.faster_rcnn.FasterRCNN(backbone,
                       num_classes=21,
                       rpn_anchor_generator=anchor_generator,
                       box_roi_pool=roi_pooler)

no need to modify the BoxHead.

[lpuglia]

@fmassa I found out what my main problem was, I was using the val set for validation only. However, to get good result on PASCAL VOC 2007 you are supposed to use trainval all together. Also, thanks to @hktxt comment I got 66% accuracy training from scratch (just 3% less than the expected). If anyone is intereseted here the highlights:

Backbone

        vgg = torchvision.models.vgg16(pretrained=True)
        backbone = vgg.features[:-1]
        for layer in backbone[:10]:
            for p in layer.parameters():
                p.requires_grad = False
        backbone.out_channels = 512
        anchor_generator = AnchorGenerator(sizes=((32, 64, 128, 256, 512),),
                                           aspect_ratios=((0.5, 1.0, 2.0),))
        roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                        output_size=7,
                                                        sampling_ratio=2)

        class BoxHead(nn.Module):
            def __init__(self, vgg):
                super(BoxHead, self).__init__()
                self.classifier = nn.Sequential(*list(vgg.classifier._modules.values())[:-1])

            def forward(self, x):
                x = x.flatten(start_dim=1)
                x = self.classifier(x)
                return x
        box_head = BoxHead(vgg)

Model

        model = torchvision.models.detection.faster_rcnn.FasterRCNN(
            backbone, #num_classes,
            rpn_anchor_generator = anchor_generator,
            box_roi_pool = roi_pooler,
            box_head = box_head,
            box_predictor = torchvision.models.detection.faster_rcnn.FastRCNNPredictor(4096, num_classes=21))

Dataset

dataset = VOCDetection(img_folder=root, year='2007', image_set='trainval', transforms=transforms)

The only aumentation i used was RandomHorizontalFlip.

Parameters

--epochs 40
--lr-steps 30
--momentum 0.9
--lr-gamma 0.1