How to get scores for every class in a prediction?

❓ How to get scores for every class in a prediction?

Hi,

This is my first time working with Detectron. I was able to successfully train Detectron2 with the YYMNIST dataset and perform object detection. https://github.com/YunYang1994/yymnist

I followed the code given in the colab notebook “Detectron2 Beginner’s Tutorial” When predicting an image, an output in this form is given: {'instances': Instances(num_instances=5, image_height=416, image_width=416, fields=[pred_boxes = Boxes(tensor([[245.0660, 358.4699, 296.7320, 410.1265], [ 20.2345, 102.6405, 47.9937, 130.1340], [ 21.0728, 21.5609, 75.9807, 76.8252], [139.2570, 246.3593, 245.1878, 362.6366], [ 4.7351, 189.9029, 83.4705, 273.8712]], device='cuda:0')), scores = tensor([0.9717, 0.9716, 0.9358, 0.8979, 0.7177], device='cuda:0'), pred_classes = tensor([7, 1, 1, 0, 2], device='cuda:0'), ])}

The detection scores, only give me a single score per predicted class. Is there a way for me to get the full softmax vector for each prediction? For example, in this format: Predicted digit: 8, Scores [0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.9, 0.02] First element in scores being the prob for digit 0, last element 9. [0-9]

Where do I have to look? I was able to obtain these vectors in the matteport implementation of Mask R-CNN using their “step-by-step prediction”-example. Is it possible to do a step-by-step prediction in Detectron2 too?

Thank you! Let me know if more info is needed.

[Jeff-Gui]

Hi, Sebastian. I guess your code may not work since score_all has not been filtered as scores, so result will not accept the faulty dimension. Below are my codes that keep score_all as a separate field in result which run successfully. By the way, I’m new to torch, so I’m not sure if the first round filtering using transformed filter_mask is the most efficient way.

def fast_rcnn_inference_single_image(
    boxes,
    scores,
    image_shape: Tuple[int, int],
    score_thresh: float,
    nms_thresh: float,
    topk_per_image: int,
):
    """
    Single-image inference. Return bounding-box detection results by thresholding
    on scores and applying non-maximum suppression (NMS).

    Args:
        Same as `fast_rcnn_inference`, but with boxes, scores, and image shapes
        per image.

    Returns:
        Same as `fast_rcnn_inference`, but for only one image.
    """
    valid_mask = torch.isfinite(boxes).all(dim=1) & torch.isfinite(scores).all(dim=1)
    if not valid_mask.all():
        boxes = boxes[valid_mask]
        scores = scores[valid_mask]

    scores = scores[:, :-1]
    num_bbox_reg_classes = boxes.shape[1] // 4
    # Convert to Boxes to use the `clip` function ...
    boxes = Boxes(boxes.reshape(-1, 4))
    boxes.clip(image_shape)
    boxes = boxes.tensor.view(-1, num_bbox_reg_classes, 4)  # R x C x 4
    
    # save the scores for later
    scores_all = scores

    # 1. Filter results based on detection scores. It can make NMS more efficient
    #    by filtering out low-confidence detections.
    filter_mask = scores > score_thresh  # R x K
    # R' x 2. First column contains indices of the R predictions;
    # Second column contains indices of classes.
    filter_inds = filter_mask.nonzero()
    if num_bbox_reg_classes == 1:
        boxes = boxes[filter_inds[:, 0], 0]
    else:
        boxes = boxes[filter_mask]
    scores = scores[filter_mask]

    # filter scores_all
    scores_all = scores_all[torch.matmul(filter_mask.cpu().byte(), torch.ByteTensor([1 for _ in range(scores_all.shape[1])])).bool()]
    
    # 2. Apply NMS for each class independently.
    keep = batched_nms(boxes, scores, filter_inds[:, 1], nms_thresh)
    if topk_per_image >= 0:
        keep = keep[:topk_per_image]
    boxes, scores, filter_inds = boxes[keep], scores[keep], filter_inds[keep]

   # index scores_all
    scores_all = scores_all[keep]

    result = Instances(image_shape)
    result.pred_boxes = Boxes(boxes)
    result.scores = scores
    result.scores_all = scores_all
    result.pred_classes = filter_inds[:, 1]
    return result, filter_inds[:, 0]

[sebastian-ruiz]

To get all the scores, replace the def fast_rcnn_inference_single_image function with this:

def fast_rcnn_inference_single_image(
    boxes,
    scores,
    image_shape: Tuple[int, int],
    score_thresh: float,
    nms_thresh: float,
    topk_per_image: int,
):
    """
    Single-image inference. Return bounding-box detection results by thresholding
    on scores and applying non-maximum suppression (NMS).

    Args:
        Same as `fast_rcnn_inference`, but with boxes, scores, and image shapes
        per image.

    Returns:
        Same as `fast_rcnn_inference`, but for only one image.
    """
    valid_mask = torch.isfinite(boxes).all(dim=1) & torch.isfinite(scores).all(dim=1)
    if not valid_mask.all():
        boxes = boxes[valid_mask]
        scores = scores[valid_mask]

    scores = scores[:, :-1]
    num_bbox_reg_classes = boxes.shape[1] // 4
    # Convert to Boxes to use the `clip` function ...
    boxes = Boxes(boxes.reshape(-1, 4))
    boxes.clip(image_shape)
    boxes = boxes.tensor.view(-1, num_bbox_reg_classes, 4)  # R x C x 4

    # save the scores for later
    scores_all = scores

    # 1. Filter results based on detection scores. It can make NMS more efficient
    #    by filtering out low-confidence detections.
    filter_mask = scores > score_thresh  # R x K
    # R' x 2. First column contains indices of the R predictions;
    # Second column contains indices of classes.
    filter_inds = filter_mask.nonzero()
    if num_bbox_reg_classes == 1:
        boxes = boxes[filter_inds[:, 0], 0]
    else:
        boxes = boxes[filter_mask]
    scores = scores[filter_mask]

    # 2. Apply NMS for each class independently.
    keep = batched_nms(boxes, scores, filter_inds[:, 1], nms_thresh)
    if topk_per_image >= 0:
        keep = keep[:topk_per_image]

    boxes, scores, filter_inds = boxes[keep], scores[keep], filter_inds[keep]

    filter = filter_inds[:, 0]
    scores_all = scores_all[filter]

    result = Instances(image_shape)
    result.pred_boxes = Boxes(boxes)
    # result.scores = scores # original code
    result.scores = scores_all
    result.pred_classes = filter_inds[:, 1]
    return result, filter_inds[:, 0]