FCOS empty box images

I am playing around with new FCOS models (thanks for that) and am encountering issues when providing images without box annotations. This is a common use case in object detection, and also works for other detector models in torchvision.

A simple example to replicate:

model = fcos_resnet50_fpn(pretrained=True)
model(torch.zeros((1,3,512,512)), targets=[{"boxes": torch.empty(0,4), "labels": torch.empty(0,1).to(torch.int64)}])

An indexing error happens in FCOSHead when running compute_loss in this part:

all_gt_classes_targets = []
all_gt_boxes_targets = []
for targets_per_image, matched_idxs_per_image in zip(targets, matched_idxs):
    gt_classes_targets = targets_per_image["labels"][matched_idxs_per_image.clip(min=0)]
    gt_classes_targets[matched_idxs_per_image < 0] = -1  # backgroud
    gt_boxes_targets = targets_per_image["boxes"][matched_idxs_per_image.clip(min=0)]
    all_gt_classes_targets.append(gt_classes_targets)
    all_gt_boxes_targets.append(gt_boxes_targets)

A workaround seems to be necessary, when having empty targets. Happy for any guidance, maybe there is also a different way necessary for me to train on empty images.

@jdsgomes @xiaohu2015 @zhiqwang

Versions

Torchvision @ master

[barschiiii]

But shouldn’t the anchor size then be adjusted manually from the multi-level feature map stride? I can see in detectron2 implementation, that they are calculating the stride each time for a backbone that is passed. Here in this case the anchor size is hardcoded, and I am wondering how these hard-coded values have been decided, the default ones seem not right to me.

[barschiiii]

SGD with AMP, and Adam without AMP seem to both run fine. Adam with AMP and gradient clipping runs into instability (NaN) issues, also with different learning rates.