Issue with Printing MAP summary in mean_ap.py

🐛 Bug

Validation step on custom dataset fails because num_classes is wrongly intialized here: https://github.com/open-mmlab/mmdetection/blob/master/mmdet/core/evaluation/mean_ap.py#L342

To Reproduce

• Follow steps in Custom Dataset preparation
• Distributed training on single GPU (Link to model config gist)

   ./tools/dist_train.sh configs/my_config_faster_rcnn_r101_fpn_1x.py 1 --validate
  

Error message (during the validate step after first epoch):

Traceback (most recent call last):
  File "./tools/train.py", line 90, in <module>
    main()
  File "./tools/train.py", line 86, in main
    logger=logger)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/mmdet-0.6.0+11e9c74-py3.6.egg/mmdet/apis/train.py", line 58, in train_detector
    _dist_train(model, dataset, cfg, validate=validate)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/mmdet-0.6.0+11e9c74-py3.6.egg/mmdet/apis/train.py", line 99, in _dist_train
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/mmcv/runner/runner.py", line 356, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/mmcv/runner/runner.py", line 272, in train
    self.call_hook('after_train_epoch')
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/mmcv/runner/runner.py", line 229, in call_hook
    getattr(hook, fn_name)(self)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/mmdet-0.6.0+11e9c74-py3.6.egg/mmdet/core/evaluation/eval_hooks.py", line 65, in after_train_epoch
    self.evaluate(runner, results)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/mmdet-0.6.0+11e9c74-py3.6.egg/mmdet/core/evaluation/eval_hooks.py", line 110, in evaluate
    print_summary=True)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/mmdet-0.6.0+11e9c74-py3.6.egg/mmdet/core/evaluation/mean_ap.py", line 327, in eval_map
    print_map_summary(mean_ap, eval_results, dataset)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/mmdet-0.6.0+11e9c74-py3.6.egg/mmdet/core/evaluation/mean_ap.py", line 370, in print_map_summary
    label_names[j], num_gts[i, j], results[j]['num_dets'],
IndexError: tuple index out of range
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/ubuntu/anaconda3/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/distributed/launch.py", line 235, in <module>
    main()
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/distributed/launch.py", line 231, in main
    cmd=process.args)

Expected Behavior

Printing the AP table:

+----------+-----+------+--------+-----------+-------+
| class    | gts | dets | recall | precision | ap    |
+----------+-----+------+--------+-----------+-------+
| class_1  | 110 | 781  | 0.091  | 0.013     | 0.002 |
| class_2  | 125 | 891  | 0.208  | 0.029     | 0.009 |
| class_3  | 98  | 1446 | 0.316  | 0.021     | 0.008 |
| class_4  | 0   | 0    | 0.000  | 0.000     | 0.000 |
| class_5  | 118 | 1578 | 0.339  | 0.025     | 0.020 |
| class_6  | 0   | 0    | 0.000  | 0.000     | 0.000 |
+----------+-----+------+--------+-----------+-------+
| mAP      |     |      |        |           | 0.019 |
+----------+-----+------+--------+-----------+-------+

Moving the line 342 after label_names is initialized, resolved the issue:

num_classes = len(label_names)

[hellock]

Thanks for the reporting! Usually L342 should work well, we will look into this bug. BTW, L342 cannot be moved after label_names being initialized, which may cause issues in L357.

[fboylu]

I am having exactly the same issue. Traceback (most recent call last): File “./tools/train.py”, line 90, in <module> main() File “./tools/train.py”, line 86, in main logger=logger) File “/datadrive/mmdetection/mmdet/apis/train.py”, line 58, in train_detector _dist_train(model, dataset, cfg, validate=validate) File “/datadrive/mmdetection/mmdet/apis/train.py”, line 99, in _dist_train runner.run(data_loaders, cfg.workflow, cfg.total_epochs) File “/data/anaconda/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/runner.py”, line 356, in run epoch_runner(data_loaders[i], **kwargs) File “/data/anaconda/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/runner.py”, line 272, in train self.call_hook(‘after_train_epoch’) File “/data/anaconda/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/runner.py”, line 229, in call_hook getattr(hook, fn_name)(self) File “/datadrive/mmdetection/mmdet/core/evaluation/eval_hooks.py”, line 65, in after_train_epoch self.evaluate(runner, results) File “/datadrive/mmdetection/mmdet/core/evaluation/eval_hooks.py”, line 110, in evaluate print_summary=True) File “/datadrive/mmdetection/mmdet/core/evaluation/mean_ap.py”, line 327, in eval_map print_map_summary(mean_ap, eval_results, dataset) File “/datadrive/mmdetection/mmdet/core/evaluation/mean_ap.py”, line 370, in print_map_summary label_names[j], num_gts[i, j], results[j][‘num_dets’], IndexError: list index out of range Traceback (most recent call last): File “/data/anaconda/envs/open-mmlab/lib/python3.7/runpy.py”, line 193, in _run_module_as_main “main”, mod_spec) File “/data/anaconda/envs/open-mmlab/lib/python3.7/runpy.py”, line 85, in _run_code exec(code, run_globals) File “/data/anaconda/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/launch.py”, line 235, in <module> main() File “/data/anaconda/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/launch.py”, line 231, in main cmd=process.args)

Thanks for lookin into the fix.