Maybe this is some problem associated with pycocotools

🐛 Bug

I am not sure if this is some problem of my own configurations, but when I run an instance segmentation problem, I received an error message.

Running per image evaluation...
Evaluate annotation type *segm*
Traceback (most recent call last):
  File "tools/train_net.py", line 171, in <module>
    main()
  File "tools/train_net.py", line 167, in main
    test(cfg, model, args.distributed)
  File "tools/train_net.py", line 104, in test
    output_folder=output_folder,
  File "/data/zhangzy/projects/play/maskrcnn-benchmark/maskrcnn_benchmark/engine/inference.py"
, line 104, in inference
    **extra_args)
  File "/data/zhangzy/projects/play/maskrcnn-benchmark/maskrcnn_benchmark/data/datasets/evalua
tion/__init__.py", line 22, in evaluate
    return coco_evaluation(**args)
  File "/data/zhangzy/projects/play/maskrcnn-benchmark/maskrcnn_benchmark/data/datasets/evalua
tion/coco/__init__.py", line 20, in coco_evaluation
    expected_results_sigma_tol=expected_results_sigma_tol,
  File "/data/zhangzy/projects/play/maskrcnn-benchmark/maskrcnn_benchmark/data/datasets/evalua
tion/coco/coco_eval.py", line 57, in do_coco_evaluation
    dataset.coco, coco_results[iou_type], file_path, iou_type
  File "/data/zhangzy/projects/play/maskrcnn-benchmark/maskrcnn_benchmark/data/datasets/evalua
tion/coco/coco_eval.py", line 287, in evaluate_predictions_on_coco
    coco_eval.evaluate()
  File "/home/zhangzy/.local/lib/python3.5/site-packages/pycocotools-2.0-py3.5-linux-x86_64.eg
g/pycocotools/cocoeval.py", line 141, in evaluate
    self._prepare()
  File "/home/zhangzy/.local/lib/python3.5/site-packages/pycocotools-2.0-py3.5-linux-x86_64.eg
g/pycocotools/cocoeval.py", line 105, in _prepare
    _toMask(gts, self.cocoGt)
  File "/home/zhangzy/.local/lib/python3.5/site-packages/pycocotools-2.0-py3.5-linux-x86_64.eg
g/pycocotools/cocoeval.py", line 93, in _toMask
    rle = coco.annToRLE(ann)
  File "/home/zhangzy/.local/lib/python3.5/site-packages/pycocotools-2.0-py3.5-linux-x86_64.eg
g/pycocotools/coco.py", line 416, in annToRLE
    rles = maskUtils.frPyObjects(segm, h, w)
  File "pycocotools/_mask.pyx", line 293, in pycocotools._mask.frPyObjects
TypeError: Argument 'bb' has incorrect type (expected numpy.ndarray, got list)

To Reproduce

Just make a small dataset with the format of coco dataset, and train on it.

Environment

Collecting environment information...
PyTorch version: 1.0.0a0+37627a1
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.4 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: version 3.12.0

Python version: 3.5
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration: 
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti
GPU 2: GeForce GTX 1080 Ti
GPU 3: GeForce GTX 1080 Ti
GPU 4: GeForce GTX 1080 Ti
GPU 5: GeForce GTX 1080 Ti
GPU 6: GeForce GTX 1080 Ti
GPU 7: GeForce GTX 1080 Ti

Nvidia driver version: 384.69
cuDNN version: Probably one of the following:
/usr/local/cuda-9.0/lib64/libcudnn.so.7.0.5
/usr/local/cuda-9.0/lib64/libcudnn_static.a

Versions of relevant libraries:
[pip3] numpy (1.14.5)
[pip3] torch (1.0.0a0+37627a1)
[pip3] torchvision (0.2.1)
[conda] Could not collect

[CoinCheung]

Sorry, this trouble is brought by our annotation coworkers. There is a annotation that the segmentation field is [793, 718, 792, 716] and the box field is [792, 716, 1, 2]. This segmentation misleads the program to execute this line, since the segmentation has only two points. I will check our data more carefully.

[CoinCheung]

I will close this issue, thanks a lot for support. @fmassa