PolyIoULoss in Oriented RCNN: ValueError The type of bboxes is notype

I tried to replace the smooth L1 loss in Oriented RCNN with PolyIoULoss but I got this error:

Traceback (most recent call last):
  File "tools/train.py", line 153, in <module>
    main()
  File "tools/train.py", line 149, in main
    meta=meta)
  File "/workspace/OBBDetection/mmdet/apis/train.py", line 129, in train_detector
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/opt/conda/envs/obbdetection/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 122, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/opt/conda/envs/obbdetection/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 32, in train
    **kwargs)
  File "/opt/conda/envs/obbdetection/lib/python3.7/site-packages/mmcv/parallel/data_parallel.py", line 31, in train_step
    return self.module.train_step(*inputs[0], **kwargs[0])
  File "/workspace/OBBDetection/mmdet/models/detectors/base.py", line 237, in train_step
    losses = self(**data)
  File "/opt/conda/envs/obbdetection/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/workspace/OBBDetection/mmdet/core/fp16/decorators.py", line 51, in new_func
    return old_func(*args, **kwargs)
  File "/workspace/OBBDetection/mmdet/models/detectors/base.py", line 172, in forward
    return self.forward_train(img, img_metas, **kwargs)
  File "/workspace/OBBDetection/mmdet/models/detectors/obb/obb_rpn.py", line 101, in forward_train
    target_bboxes_ignore)
  File "/workspace/OBBDetection/mmdet/models/dense_heads/base_dense_head.py", line 54, in forward_train
    losses = self.loss(*loss_inputs, gt_bboxes_ignore=gt_bboxes_ignore)
  File "/workspace/OBBDetection/mmdet/models/dense_heads/obb/oriented_rpn_head.py", line 81, in loss
    gt_bboxes_ignore=gt_bboxes_ignore)
  File "/workspace/OBBDetection/mmdet/core/fp16/decorators.py", line 131, in new_func
    return old_func(*args, **kwargs)
  File "/workspace/OBBDetection/mmdet/models/dense_heads/obb/obb_anchor_head.py", line 515, in loss
    num_total_samples=num_total_samples)
  File "/workspace/OBBDetection/mmdet/core/utils/misc.py", line 54, in multi_apply
    return tuple(map(list, zip(*map_results)))
  File "/workspace/OBBDetection/mmdet/models/dense_heads/obb/obb_anchor_head.py", line 445, in loss_single
    avg_factor=num_total_samples)
  File "/opt/conda/envs/obbdetection/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/workspace/OBBDetection/mmdet/models/losses/obb/poly_iou_loss.py", line 216, in forward
    **kwargs)
  File "/workspace/OBBDetection/mmdet/models/losses/utils.py", line 94, in wrapper
    loss = loss_func(pred, target, **kwargs)
  File "/workspace/OBBDetection/mmdet/models/losses/obb/poly_iou_loss.py", line 113, in poly_giou_loss
    areas1, areas2 = get_bbox_areas(pred), get_bbox_areas(target)
  File "/workspace/OBBDetection/mmdet/core/bbox/transforms_obb/misc.py", line 50, in get_bbox_areas
    raise ValueError('The type of bboxes is notype')
ValueError: The type of bboxes is notype

It is triggered in this code block at mmdet/code/bbox/transforms_obb/misc.py

def get_bbox_areas(bboxes):
    btype = get_bbox_type(bboxes)
    if btype == 'hbb':
        wh = bboxes[..., 2:] - bboxes[..., :2]
        areas = wh[..., 0] * wh[..., 1]
    elif btype == 'obb':
        areas = bboxes[..., 2] * bboxes[..., 3]
    elif btype == 'poly':
        pts = bboxes.view(*bboxes.size()[:-1], 4, 2)
        roll_pts = torch.roll(pts, 1, dims=-2)
        xyxy = torch.sum(pts[..., 0] * roll_pts[..., 1] -
                         roll_pts[..., 0] * pts[..., 1], dim=-1)
        areas = 0.5 * torch.abs(xyxy)
    else:
        raise ValueError('The type of bboxes is notype')

    return areas

[jbwang1997]

you can refer to RetinaNet to decode prediction and target boxes in training

Do I just need to insert reg_decoded_bbox=True, here?

Yes, reg_decoded_bbox=True will decode predction and target boxes.

[ReusJeffery]

Oh, I haven’t tried PolyIoULoss in Oriented RPN. Let me do some research. Hello,did you solve the NaN probelm?