How to get the evaluations

❓

Hello, I already trained my own dataset that is in coco format, but it has different classes.

If I understand correctly I need to set MetadataCatalog.get(dataset_name).evaluator_type

and I try to run the following code to get the evaluations

MetadataCatalog.get("dataset_name").evaluator_type="coco"
cfg.DATASETS.TEST = ("dataset_name",)  
cfg.MODEL.WEIGHTS = os.path.join("output/model_final.pth")
model = DefaultTrainer.build_model(cfg)
DetectionCheckpointer(model, save_dir=cfg.OUTPUT_DIR).resume_or_load(cfg.MODEL.WEIGHTS)
res = DefaultTrainer.test(cfg, model)

But I got this error NotImplementedError: Please either implement build_evaluator() in subclasses, or pass your evaluator as arguments to DefaultTrainer.test().

Thank you for your attention.

[insanesac]

I could make it work by being sure the train/test split was the same as the one I used during training. The easiest was to evaluate in the same script. Using V7labs I could pull and split several times (GPU and local machine) and that was the reason I had only 0.0 as AP, because the test on my local machine was not the same as the one on GPU

I have separate train and test datasets. It works without any issue if I do the evaluation in the same script as training. But when I try to evaluate in a separate script after loading my model, I get empty APs.

[djpecot]

For anyone using Colab, if you put this in at the front of your code with the rest of the detectron installs, seems to do the trick: !pip install -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'