Evaluation on coco2017 (5000 images) is extremely slow

❓ Questions and Help

Hi, I found that evaluation on coco2017 with 5000 images is extremely slow.

I haven’t finished the evaluation process yet, but it seems that this would take about 3 hours to complete.

This is the command I used on 1 gpu,

python tools/test_net.py --config-file 'configs/e2e_mask_rcnn_R_50_FPN_1x.yaml'  TEST.IMS_PER_BATCH 4

And I found that the gpu usage is zero, this is quite weird.

I did not change the parameter MODEL.ROI_HEADS.DETECTIONS_PER_IMG. Could you help me figure it out?

[chengyangfu]

If you are using ‘configs/e2e_mask_rcnn_R_50_FPN_1x.yaml’ for testing without specifying new MODEL.WEIGHT, it means you use an untrained model for inference. So, you will get very bad detection results. It also will be very slow because there is some post-processing after CNN inference. For example, using some threshold to filter out low confidence predictions. In your case, because you didn’t train the model and so the thresholding does not work properly.

If you just want to test the detection, you can use the script in config/caffe2, in this case, the program will download the trained detection model from facebook server automatically and run the test.
Otherwise, you need to train the model first. Then use your trained model for inference by during adding MODEL.WEIGHT YOURMODEL in your testing command.

[sarahmass]

I have been using WEIGHT: “https://download.pytorch.org/models/maskrcnn/e2e_mask_rcnn_R_50_FPN_1x.pth” as my weights and it takes less than 20 minutes to run inference on all 5000 images. I don’t have the exact time because I am running it on AzureML and I have to build the libraries each time I do my tests. I am also running on a single gpu with TEST.IMS_PER_BATCH: 10. I hope this helps.