"The testing results of the whole dataset is empty" error only on test time

Hi! I am using mmdet for a custom dataset. The object detection task I am working with is a single-class problem, and I am only concerned with predicting the bounding box for this class of objects. I am facing two problems.

1. At training time, the loss decreases and the accuracy increases, not showing any losses of nan, so I know the training has been done properly. At validation after every epoch, I also get a decent mAP.

But when I test the model with dist_test.sh, I always get the following error:

2022-11-24 11:52:27,581 - mmdet - INFO - Evaluating bbox...
Loading and preparing results...
2022-11-24 11:52:27,581 - mmdet - ERROR - The testing results of the whole dataset is empty.
OrderedDict([('rpn_bbox_mAP', -1.0), ('rpn_bbox_mAP_50', -1.0), ('rpn_bbox_mAP_75', -1.0), ('rpn_bbox_mAP_s', -1.0), ('rpn_bbox_mAP_m', -1.0), ('rpn_bbox_mAP_l', -1.0), ('rpn_bbox_mAP_copypaste', '-1.000 -1.000 -1.000 -1.000 -1.000 -1.000')])

I have looked into other discussion posts and issues submitted, from which I know that the issue cannot be due to:

• a high learning rate (verified that lr=0.001 does well with the training)
• classes not aligned properly (changed all num_classes to 1 as needed)
• img_prefix set improperly (checked that img_prefix is indeed set properly)

When running dist_test.sh command, here’s an example output:

----- info -----
bboxes: []
img_id: 15
result: [array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32)]
label: 77
self.cat_ids: []

----- info -----
bboxes: []
img_id: 15
result: [array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32)]
label: 78
self.cat_ids: []

----- info -----
bboxes: []
img_id: 15
result: [array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32), array([], shape=(0, 5), dtype=float32)]
label: 79
self.cat_ids: []

The empty self.cat_ids and multiple iterations of the same image even when there is a single bounding box were suspicious, so I have also looked into the mmdet codebase, in particular into mmdet/datasets/coco.py where the output above is generated. After manually adding the print statements, I found out that self.cat_ids is always [], where bounding boxes, when they are successfully returned, have five values:

bboxes: [[0.0000000e+00 1.3298181e+02 2.9212219e+01 1.8850955e+02 6.2224197e-01]
 [2.2335358e+02 1.6708778e+02 2.5429608e+02 2.2586888e+02 5.8386362e-01]
 [2.3871152e+02 3.2964802e-01 2.6771085e+02 2.0484722e+01 4.6579489e-01]
 [3.8837106e+02 2.5154613e+02 4.1521347e+02 3.0727646e+02 4.0440291e-01]
 [2.5092139e+02 3.8236469e+02 2.8243182e+02 4.1426154e+02 3.2358450e-01]
 [3.5458630e+02 1.8993091e+02 4.1600000e+02 3.0106595e+02 1.1559867e-01]
 [5.0709690e+01 2.1426732e+02 1.3901105e+02 3.7354681e+02 1.0578756e-01]
 [1.1068954e+00 3.8645844e+02 3.1745728e+01 4.1321161e+02 8.0695361e-02]
 [2.2867912e+02 3.3605237e+02 2.9344427e+02 4.1365192e+02 7.1543925e-02]
 [1.0791272e+02 3.0769669e+02 1.6822672e+02 3.4960214e+02 6.8247803e-02]
 [1.7628798e+02 2.8829105e+02 2.0807918e+02 3.1405154e+02 5.7077557e-02]
 [2.0341975e+02 1.3263596e+02 3.0045563e+02 2.4642096e+02 5.6500241e-02]
 [0.0000000e+00 5.6556923e+01 6.8761665e+01 1.8821082e+02 5.0539084e-02]]

As far as I know, bounding boxes only have four entries (either xyxy or xywh), so where would the fifth element, a very small value all less than 1, be coming from?

1. When I enable multiple gpus (say, 2 or 4), I always get an error message in the middle and the training halts. The error message isn’t very informative, and it just seems like it’s a multithreading issue. The issue is cleared when I try with 1 gpu. From a discussion post, I saw that this might be resolved by changing nccl to gloo backend, but with no luck. Has anyone encountered this issue before, and perhaps resolved it?

Thanks!

[JiHunWang]

Thank you so much for your detailed response! Based on your suggestion, I took another look at the codebase and how things are set up on my part.

• len(result) was returning 80
• self.cat_ids was [], an empty array
• In the second Shared2FCBBoxHead, fc_out_channels=1024 (I think this one is set properly)

The only reasonable conclusion that I could make with this was that the custom dataset was not loaded properly. This was indeed the case, since:

self.coco = COCO(ann_file)
self.cat_ids = self.coco.get_cat_ids()
self.cat2label = {cat_id: i for i, cat_id in enumerate(self.cat_ids)}
print(self.cat2label)

was returning {}. The reason behind this is probably because I did not use CustomDataset but sticked to PointCocoDataset and supplied a separate information about classes. For some reason, I could not get CustomDataset working yet, but a temporary workaround for this was setting self.cat2label = {0: 0}.

For a subset of the test dataset, I get the following result now:

----- info -----
bbox: [228.740234375, 162.5919647216797, 24.416778564453125, 72.21844482421875]
img_id: 1
score: 0.8173308372497559
self.cat_ids: 0


----- info -----
bbox: [3.4858903884887695, 121.35139465332031, 24.06021785736084, 72.70675659179688]
img_id: 1
score: 0.7843648791313171
self.cat_ids: 0


----- info -----
bbox: [248.17266845703125, 381.075439453125, 29.7132568359375, 31.760009765625]
img_id: 1
score: 0.5045464038848877
self.cat_ids: 0


----- info -----
bbox: [383.8284912109375, 241.9992218017578, 31.60894775390625, 70.07493591308594]
img_id: 1
score: 0.4161221981048584
self.cat_ids: 0

This is as what I have expected, since I have only one class, and the image with img_id=1 has four annotations, listed below:

        {
            "id": "1_0",
            "image_id": 1,
            "category_id": 0,
            "area": 1629.9153974931257,
            "bbox": [
                228.48628206109058,
                164,
                24.695687840804936,
                66
            ],
            "iscrowd": 0,
            "point": [
                248,
                216,
                248,
                216
            ]
        },
        {
            "id": "1_1",
            "image_id": 1,
            "category_id": 0,
            "area": 2002,
            "bbox": [
                0,
                121,
                26,
                77
            ],
            "iscrowd": 0,
            "point": [
                26,
                179,
                26,
                179
            ]
        },
        {
            "id": "1_2",
            "image_id": 1,
            "category_id": 0,
            "area": 1705,
            "bbox": [
                247,
                361,
                31,
                55
            ],
            "iscrowd": 0,
            "point": [
                272,
                409,
                272,
                409
            ]
        },
        {
            "id": "1_3",
            "image_id": 1,
            "category_id": 0,
            "area": 2819.0602749068084,
            "bbox": [
                376.11363713652656,
                239.32270398892632,
                39.88636286347344,
                70.67729601107368
            ],
            "iscrowd": 0,
            "point": [
                412,
                291,
                412,
                291
            ]
        }

As a result, the evaluation result I get on evaluation with dist_test.sh now looks pretty good!

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.379
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.803
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.310
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.153
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.398
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.477
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.456
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.456
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.456
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.280
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.477
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.515

Thank you so much for your help. Let me know if there’s anything else you would like to comment on!

[RangiLyu]

I am running Group R-CNN that uses mmdet.

@jshilong Please take a look.