Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

pretraining Detectron2 FPN models

See original GitHub issue

Hi Lightly-team,

Is there a way to use Lightly for self-supervised pretraining of Detectron2 models? I think that would be valuable for many people. The “Lightly Backbone Playground.ipynb” colab shows the concept of using a Lightly implemented SimCLR to pretrain a torchvision resnet50. However, the most effective Detectron2 models are resnet-FPNs (Feature Pyramid Networks).

The detectron standard Faster RCNN with resnet-FPN contains a backbone: detmodel.backbone a RPN (region proposal network): detmodel.proposal_generator and a ROI (region of interest): detmodel.head roi_heads The backbone itself is a resnet-FPN and I assume it is easiest to train the bottom-up part of the resnet-FPN ( detmodel.backbone.bottom_up), as that is basically a resnet.

It seems doable except that the feature-shapes are mismatched. I created a minimal working example in colab:

Is there a way to get the SimCLR working with this backbone?

The colab above contains a runnable example, but I tried to capture the essence in the code below:

# Get original resnet model
resnet50 = torchvision.models.resnet50()
# Remove linear head
rbackbone = nn.Sequential(*list(resnet50.children())[:-1],)

# Toy input
toy_input_view_a = torch.randn((11,3,256,256)).to(device)

# Output shape of original resnet model
np.shape( rbackbone(toy_input_view_a) )
# Out: torch.Size([11, 2048, 1, 1])

# In the forward function of the file lightly/models/
# f0 = self.backbone(x0).flatten(start_dim=1)
# which leads to the shape torch.Size([11, 2048]) for rbackbone's output

#However, with the detectron model
detmodel = build_model(<config_file>)
backbone = nn.Sequential(*list(detmodel.backbone.bottom_up.children()),)

# We get a different output shape to start with
np.shape( backbone(toy_input_view_a) )
# Out: torch.Size([11, 2048, 8, 8])

# Which, after the flatten in lightly simclr model lightly/models/
# leads to the shape torch.Size([11, 131072])

# Thus 
simCLR = lightly.models.SimCLR(backbone, num_ftrs=2048)
out = simCLR(toy_input_view_a, toy_input_view_a)
# Leads to the error
# RuntimeError: mat1 and mat2 shapes cannot be multiplied (11x131072 and 2048x2048)

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:7 (4 by maintainers)

github_iconTop GitHub Comments

IgorSusmeljcommented, Nov 3, 2021

We are working on a new tutorial to address this. Hopefully, this will be out next week!

  • How to pretrain a detectron2 model using Lightly
  • How to use the pretrained model to fine-tune a detectron2 model
philippmwirthcommented, Oct 22, 2021

Here’s an example. In the code for creating the backbone in the MoCoModel, we use

backbone = nn.Sequential(
    *list(resnet.children())[:-1], # output is [bsz, 2048, 8, 8]
    nn.AdaptiveAvgPool2d(1), # output is [bsz, 2048]
Read more comments on GitHub >

github_iconTop Results From Across the Web

Detectron2 Model Zoo and Baselines
Detectron2 Model Zoo and Baselines. Introduction. This file documents a large collection of baselines trained with detectron2 in Sep-Oct, 2019.
Read more >
MODEL.ANCHOR_GENERATOR.NAME . class detectron2.modeling. FPN (bottom_up, in_features ... pretrain_img_size (int) – input image size for pretraining models.
Read more >
Tutorial 6: Pre-train a Detectron2 Backbone with Lightly
In this tutorial we show how you can do self-supervised pre-training of a Detectron2 backbone with lightly. The focus of this tutorial is...
Read more >
Training an Object Detection Model in a few minutes using ...
Detectron2 includes a variety of models like Faster R-CNN, Mask R-CNN, RetinaNet, DensePose, Cascade R-CNN, Panoptic FPN, and TensorMask.
Read more >
Train a Custom Object Detector with Detectron2 and FiftyOne
Combine the dataset curation of FiftyOne with the model training of ... Detectron2 tutorial, we now fine-tune a COCO-pretrained R50-FPN Mask ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found