No results on test dataset

• Operating system: ubuntu 16.04
• CUDA version: 9.0
• cuDNN version: 7.0.5
• NVIDIA driver version: 396.44
• GPU models (for all devices if they are not all the same): 950M 4Gb
• python --version output: 2.7.12

I trained a Faster R-CNN with ResNet50 model on my custom train dataset, and everything seems ok. But when I test it on my custom test dataset (test of train_net.py), I get this results:

INFO test_engine.py: 320: Wrote detections to: home/juan/detectron/test/tools_test/generalized_rcnn/detections.pkl INFO test_engine.py: 162: Total inference time: 101.063s INFO task_evaluation.py: 76: Evaluating detections INFO json_dataset_evaluator.py: 162: Writing bbox results json to: home/juan/detectron/test/tools_test/generalized_rcnn/bbox_tools_test_results.json INFO task_evaluation.py: 62: Evaluating bounding boxes is done! INFO task_evaluation.py: 181: copypaste: Dataset: tools_test INFO task_evaluation.py: 183: copypaste: Task: box INFO task_evaluation.py: 186: copypaste: AP,AP50,AP75,APs,APm,APl INFO task_evaluation.py: 187: copypaste: -1.0000,-1.0000,-1.0000,-1.0000,-1.0000,-1.0000

But If I run infer_simple.py with the test images, I can see that model is relatively good:

If I test it on my train dataset (I know that this isn’t correct), I get this:

INFO test_engine.py: 320: Wrote detections to: /home/juan/detectron/test/tools_train/generalized_rcnn/detections.pkl INFO test_engine.py: 162: Total inference time: 180.051s INFO task_evaluation.py: 76: Evaluating detections INFO json_dataset_evaluator.py: 162: Writing bbox results json to: /home/juan/detectron/test/tools_train/generalized_rcnn/bbox_tools_train_results.json Loading and preparing results… DONE (t=0.01s) creating index… index created! Running per image evaluation… Evaluate annotation type bbox DONE (t=0.38s). Accumulating evaluation results… DONE (t=0.05s). INFO json_dataset_evaluator.py: 222: ~~~~ Mean and per-category AP @ IoU=[0.50,0.95] ~~~~ INFO json_dataset_evaluator.py: 223: 93.0 INFO json_dataset_evaluator.py: 231: 95.5 INFO json_dataset_evaluator.py: 231: 91.5 INFO json_dataset_evaluator.py: 231: 91.9 INFO json_dataset_evaluator.py: 232: ~~~~ Summary metrics ~~~~ Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.930 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.997 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.993 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.908 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.938 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.449 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.949 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.951 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.927 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.958 INFO json_dataset_evaluator.py: 199: Wrote json eval results to: ./test/tools_train/generalized_rcnn/detection_results.pkl INFO task_evaluation.py: 62: Evaluating bounding boxes is done! INFO task_evaluation.py: 181: copypaste: Dataset: tools_train INFO task_evaluation.py: 183: copypaste: Task: box INFO task_evaluation.py: 186: copypaste: AP,AP50,AP75,APs,APm,APl INFO task_evaluation.py: 187: copypaste: 0.9298,0.9967,0.9932,-1.0000,0.9079,0.9380

I don’t know why if model can detect object on test dataset, I don’t get summary metrics… and only I get -1.0000 in all metrics.

My original dataset’s labels was in xml, and I convert it to json with this file, but the generated json to test and train are slightly differents, because this script assign a id number to a name in order in which script find it in dataset. I testes withouth changes, change the json with find+replace to leave category_id same in both json and finally, I changed script, I replaced line 22, and I assign category_item_id to harcoded name:

if name == "plier":
    category_item_id = 1
elif name == "screwdriver":
    category_item_id = 2
elif name == "wrench":
    category_item_id = 3

But in all cases, I always get same results if I test on my test dataset.

[gadcam]

Looks like a duplicate of #293. https://github.com/facebookresearch/Detectron/issues/293#issuecomment-397962141 could be the solution

(#588 could be relevant too)

[juanluisdb]

The folder can’t contain “test”