Keypoint RCNN Evaluation

❓ Questions and Help

I am unsure if this is more of a usage issue or an actual problem. Currently I am training on top of Keypoint RCNN to only detect 6 keypoints (whereas the baseline model I am using has 17 keypoints). The model is able to train fine. Whenever I use the cocoevaluator, it is able to create AP statistics for bounding boxes, but fails upon doing keypoint analysis with the following error:

/usr/local/lib/python3.6/dist-packages/pycocotools/cocoeval.py in computeOks(self, imgId, catId)
    220                 if k1>0:
    221                     # measure the per-keypoint distance if keypoints visible
--> 222                     dx = xd - xg
    223                     dy = yd - yg
    224                 else:

ValueError: operands could not be broadcast together with shapes (17,) (6,) 

I am guessing that there was some configuration variable I failed to set appropriately that would dictate that the new # of keypoints is 6 (instead of 17). But I feel I did that by setting cfg.MODEL.ROI_KEYPOINT_HEAD.NUM_KEYPOINTS = 6. Here is my full config:

from detectron2.engine import DefaultTrainer
from detectron2.config import get_cfg

cfg = get_cfg()
cfg.merge_from_file("./detectron2_repo/configs/COCO-Keypoints/keypoint_rcnn_R_50_FPN_3x.yaml")
cfg.DATASETS.TRAIN = ("spine-segmentation_train",)
cfg.DATASETS.TEST = ("spine-segmentation_val",)   # no metrics implemented for this dataset
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = "detectron2://COCO-Keypoints/keypoint_rcnn_R_50_FPN_3x/137849621/model_final_a6e10b.pkl"  # initialize from model zoo
cfg.SOLVER.IMS_PER_BATCH = 2

cfg.SOLVER.BASE_LR = 0.00025
cfg.SOLVER.MAX_ITER = 600   
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128  
cfg.MODEL.ROI_KEYPOINT_HEAD.NUM_KEYPOINTS = 6
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1  # only has one class (spine)

And here is the evaluation code

from detectron2.evaluation import COCOEvaluator, inference_on_dataset
from detectron2.data import build_detection_test_loader
evaluator = COCOEvaluator("spine-segmentation_val", cfg, False, output_dir="./output/")
val_loader = build_detection_test_loader(cfg, "spine-segmentation_val")
inference_on_dataset(trainer.model, val_loader, evaluator)

My general question is how do I set up detectron to evaluate the keypoint Rcnn base model? is there anything else I need to do on the configuration front?

[abhisuri97]

Nevermind, figured it out! For future reference, it just needs to be an array of floats that is the same length as the number of keypoints you’re looking for. Refer to section 1.3 here on how to get those numbers

[jianchaoci]

COCO API requires you to specify the OKS for each keypoint. You can specify this using cfg.TEST.KEYPOINT_OKS_SIGMAS when your keypoints are different from COCO.

Hi, I also have some confusions of calculating the sigmas. According to the instruction, σi2=E[di2/s2]. But I am not sure is the ‘s’ means the object ‘area’? or the sqrt of area explained in some blogs.