How to detect the image by trained model based on DemoPredictor?

Instructions To Reproduce the 🐛 Bug:

I want to use the trained model to detect and I saw the instruction of jupyter-notebook that it used DemoPredictor to make a predictor. However, it cannot work (The results were all empty.) if I attempt to individually load the model rather than predict directly after I finished the training. If I change the DemoPredictor to default predict from Detectron2, then it can work.

Here is my simple main code

import cv2, json
from matplotlib import pyplot as plt
from d2go.utils.demo_predictor import DemoPredictor
from detectron2.utils.visualizer import ColorMode
from detectron2.engine import DefaultPredictor
from d2go.runner import Detectron2GoRunner
from d2go.model_zoo import model_zoo
import os
from detectron2.data import MetadataCatalog, DatasetCatalog
from detectron2.data.datasets import register_coco_instances

register_coco_instances("my_dataset_val", {}, "test.json", "./test_Images/")
my_dataset_val_metadata = MetadataCatalog.get("my_dataset_val")
val_dataset_dicts = DatasetCatalog.get("my_dataset_val")
MetadataCatalog.get("my_dataset_val").set(thing_classes=['top', 'bottom'])

yaml_file = 'mask_rcnn_fbnetv3a_C4.yaml'

runner = Detectron2GoRunner()
cfg = runner.get_default_cfg()
cfg.merge_from_file(model_zoo.get_config_file(yaml_file))
cfg.DATALOADER.NUM_WORKERS = 2
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128   # faster, and good enough for this toy dataset
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 4  # only has one class (ballon)
# load weights
cfg.MODEL.WEIGHTS = 'model_final.pth'
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7   # set the testing threshold for this model
# Set training data-set path
cfg.DATASETS.TEST = ("my_dataset_val", )

# predictor = DefaultPredictor(cfg)

model = runner.build_model(cfg)
predictor = DemoPredictor(model)

for d in val_dataset_dicts: 
    im = cv2.imread(d["file_name"])
    outputs = predictor(im)
    print(outputs)

If I use this one to make a predictor, then it can load the weight and predict correctly.

predictor = DefaultPredictor(cfg)

However, if I change to this one which was from https://github.com/facebookresearch/d2go/blob/master/demo/d2go_beginner.ipynb

model = runner.build_model(cfg)
predictor = DemoPredictor(model)

Then it seems to fail to load the model that outputs are nothing.

Or should I need to use this way to load the model_final.pth and to do inference?

from mobile_cv.predictor.api import create_predictor
patch_d2_meta_arch()

cfg_name = 'faster_rcnn_fbnetv3a_dsmask_C4.yaml'
pytorch_model = model_zoo.get(cfg_name, trained=True)
pytorch_model.cpu()

with create_fake_detection_data_loader(224, 320, is_train=False) as data_loader:
    predictor_path = convert_and_export_predictor(
            model_zoo.get_config(cfg_name),
            copy.deepcopy(pytorch_model),
            "torchscript_int8@tracing",
            './',
            data_loader,
        )

model = create_predictor(predictor_path)

Expected behavior:

I expect I can load the model_final.pth to output the correct results.

Thank you so much!

[chiehpower]

Hi @zhanghang1989 ,

Thanks for your reply! I was training the model by this config file mask_rcnn_fbnetv3a_C4.yaml that one was from here.

The training code was taken from here: https://github.com/facebookresearch/d2go/blob/master/demo/d2go_beginner.ipynb.

Thank you so much

[zhanghang1989]

Sorry for the delayed reply.

I see. Looks like you’re using it with a local model. The model weight is not loaded. Just add the following code before creating the predictor:

checkpointer = runner.build_checkpointer(cfg, model, save_dir=output_dir)
checkpoint = checkpointer.resume_or_load(cfg.MODEL.WEIGHTS, resume=resume)