Object are localized but not classified

❓ Questions and Help

I fine-tuned Faster R-CNN with backbone FPN-50 on Cityscapes dataset. Objects can be localized accurately but are always classified incorrectly, such as: all ‘car’ => ‘bus’, ‘bus’ => ‘car’ , ‘person’ => ‘truck’, ‘bicycle’ => ‘motorcycle’, etc. predictions.get_field(“labels”) is always incorrect. I don’t know what’s wrong. Please help me!

I fine-tune the network as follow:

1. Remove layers having ‘cls_score’ and ‘bbox_pred’, and save model:

model = build_detection_model(cfg)
device = torch.device(cfg.MODEL.DEVICE)
model.to(device)
optimizer = make_optimizer(cfg, model)
scheduler = make_lr_scheduler(cfg, optimizer)
save_to_disk = get_rank() == 0
checkpointer = DetectronCheckpointer(cfg, model, optimizer, scheduler, cfg.OUTPUT_DIR, save_to_disk)
checkpointer.load(cfg.MODEL.WEIGHT, init=True)
checkpointer.save(path_output)

maskrcnn_benchmark/utils/checkpoint.py is modified as:

class Checkpointer(object):
    def load(self, f=None, init=False):
        if self.has_checkpoint():
            # override argument with existing checkpoint
            f = self.get_checkpoint_file()
        if not f:
            # no checkpoint could be found
            self.logger.info("No checkpoint found. Initializing model from scratch")
            return {}
        self.logger.info("Loading checkpoint from {}".format(f))
        checkpoint = self._load_file(f)
        if init:
            checkpoint['model'] = {k.replace('module.', ''): v for k, v in checkpoint['model'].items()
                                    if 'cls_score' not in k and 'bbox_pred' not in k and 'mask_fcn_logits' not in k}
        self._load_model(checkpoint)
        if "optimizer" in checkpoint and self.optimizer:
            self.logger.info("Loading optimizer from {}".format(f))
            self.optimizer.load_state_dict(checkpoint.pop("optimizer"))
        if "scheduler" in checkpoint and self.scheduler:
            self.logger.info("Loading scheduler from {}".format(f))
            self.scheduler.load_state_dict(checkpoint.pop("scheduler"))

1. Change config file: NUM_CLASSES: 8, for background and 7 object classes
2. Load model and train as normal. Here is loss:

[sandeep-d-patil]

Hey @luisdecker you can check out my repo, have tried to make a cityscapes dataset, similar to matterport repo. Hope this helps!

[tuanhanhdmprof]

@tuanhanhdmprof mình đã cố gắng sử dụng file convert này https://github.com/facebookresearch/Detectron/blob/master/tools/convert_cityscapes_to_coco.py nhưng vẫn không thành công. Bạn có thể chỉ rõ cho mình nên sắp xếp thư mục dataset như thế nào được không?

Mình nhầm ý trên. Hướng dẫn để convert nhé:

1. Download file json của cityscapes về.
2. Download / pip install cityscapesScripts.
3. tổ chức thư mục: – root ---- gtFine_trainvaltest ---- Detectron ---- output
4. run lệnh sau: python ./Detectron/tools/convert_cityscapes_to_coco.py --dataset cityscapes_instance_only --datadir . --outdir ./output/

cái thứ 3 cấu trúc lại thì sửa lại lệnh run trên là ok