pretraining Detectron2 FPN models

Hi Lightly-team,

Is there a way to use Lightly for self-supervised pretraining of Detectron2 models? I think that would be valuable for many people. The “Lightly Backbone Playground.ipynb” colab shows the concept of using a Lightly implemented SimCLR to pretrain a torchvision resnet50. However, the most effective Detectron2 models are resnet-FPNs (Feature Pyramid Networks).

The detectron standard Faster RCNN with resnet-FPN contains a backbone: detmodel.backbone a RPN (region proposal network): detmodel.proposal_generator and a ROI (region of interest): detmodel.head roi_heads The backbone itself is a resnet-FPN and I assume it is easiest to train the bottom-up part of the resnet-FPN ( detmodel.backbone.bottom_up), as that is basically a resnet.

It seems doable except that the feature-shapes are mismatched. I created a minimal working example in colab: https://colab.research.google.com/drive/11AIdCGkHC1loQjvQnj51kXCD9TKNAib5?usp=sharing

Is there a way to get the SimCLR working with this backbone?

The colab above contains a runnable example, but I tried to capture the essence in the code below:

# Get original resnet model
resnet50 = torchvision.models.resnet50()
# Remove linear head
rbackbone = nn.Sequential(*list(resnet50.children())[:-1],)

# Toy input
toy_input_view_a = torch.randn((11,3,256,256)).to(device)

# Output shape of original resnet model
np.shape( rbackbone(toy_input_view_a) )
# Out: torch.Size([11, 2048, 1, 1])

# In the forward function of the file lightly/models/simclr.py:88
# f0 = self.backbone(x0).flatten(start_dim=1)
# which leads to the shape torch.Size([11, 2048]) for rbackbone's output

#However, with the detectron model
detmodel = build_model(<config_file>)
backbone = nn.Sequential(*list(detmodel.backbone.bottom_up.children()),)

# We get a different output shape to start with
np.shape( backbone(toy_input_view_a) )
# Out: torch.Size([11, 2048, 8, 8])

# Which, after the flatten in lightly simclr model lightly/models/simclr.py:88
# leads to the shape torch.Size([11, 131072])

# Thus 
simCLR = lightly.models.SimCLR(backbone, num_ftrs=2048)
out = simCLR(toy_input_view_a, toy_input_view_a)
# Leads to the error
# RuntimeError: mat1 and mat2 shapes cannot be multiplied (11x131072 and 2048x2048)

[IgorSusmelj]

We are working on a new tutorial to address this. Hopefully, this will be out next week!

• How to pretrain a detectron2 model using Lightly
• How to use the pretrained model to fine-tune a detectron2 model

[philippmwirth]

Here’s an example. In the code for creating the backbone in the MoCoModel, we use

backbone = nn.Sequential(
    *list(resnet.children())[:-1], # output is [bsz, 2048, 8, 8]
    nn.AdaptiveAvgPool2d(1), # output is [bsz, 2048]
)