Training Mask R-CNN using RLE bitmasks error

I am following the Mask R-CNN tutorial and changed the dataset_dict to support segmentation maps in bitmap format using RLE instead of polygons. I confirmed the data is processed properly using detectron2 visualization tool. When trying to train the network, I’m getting an error regarding polygons.

File "/home/ubuntu/detectron2/detectron2/data/detection_utils.py", line 149, in transform_instance_annotations polygons = [np.asarray(p).reshape(-1, 2) for p in annotation["segmentation"]] File "/home/ubuntu/detectron2/detectron2/data/detection_utils.py", line 149, in <listcomp> polygons = [np.asarray(p).reshape(-1, 2) for p in annotation["segmentation"]] ValueError: cannot reshape array of size 1 into shape (2)

It seems there is no support for RLE format during training although the visualization works. Is there any way to train using bitmasks?

[siyuanfeng-tri]

The above issue about evaluation is hopefully fixed in 6901cc7

I think I need to do a bit more to get this actually working.

1. I had to cast this into int explicitly, otherwise json dump complains about TypeError: Object of type 'uint32' is not JSON serializable https://github.com/facebookresearch/detectron2/blob/master/detectron2/data/datasets/coco.py#L339
2. I had to decode segmentation["counts"] into an ascii string otherwise json dump complains about not being able to serialize bytes. TypeError: Object of type 'bytes' is not JSON serializable

[ppwwyyxx]

Now the default dataloader can work with RLE formats inside your dataset. All you need is:

1. use RLE format (documented at https://detectron2.readthedocs.io/tutorials/datasets.html#standard-dataset-dicts) in your dataset
2. set INPUT.MASK_FORMAT='bitmask'.