Issues training with CMU_Panoptic

Hello,

1. I am trying to train the model starting from pretrained resent on the cmu_panoptic dataset. However, I get the following error:

Traceback (most recent call last):
  File "HumanObj_videos_ResNet/train.py", line 277, in <module>
    main()
  File "HumanObj_videos_ResNet/train.py", line 273, in main
    trainer.train()
  File "HumanObj_videos_ResNet/train.py", line 77, in train
    self.train_epoch(epoch)
  File "HumanObj_videos_ResNet/train.py", line 192, in train_epoch
    for iter_index, meta_data in enumerate(self.loader):
  File "/z/home/mkhoshle/env/romp2/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/z/home/mkhoshle/env/romp2/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/z/home/mkhoshle/env/romp2/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/z/home/mkhoshle/env/romp2/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/z/home/mkhoshle/env/romp2/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/z/home/mkhoshle/env/romp2/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/z/home/mkhoshle/env/romp2/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/z/home/mkhoshle/Human_object_transform/HumanObj_videos_ResNet/lib/dataset/mixed_dataset.py", line 79, in __getitem__
    annots = self.datasets[dataset_id][index_sample]
  File "/z/home/mkhoshle/Human_object_transform/HumanObj_videos_ResNet/lib/dataset/image_base.py", line 375, in __getitem__
    return self.get_item_single_frame(index)
  File "/z/home/mkhoshle/Human_object_transform/HumanObj_videos_ResNet/lib/dataset/image_base.py", line 123, in get_item_single_frame
    kp3d, valid_masks[:,1] = self.process_kp3ds(info['kp3ds'], used_person_inds, \
  File "/z/home/mkhoshle/Human_object_transform/HumanObj_videos_ResNet/lib/dataset/image_base.py", line 284, in process_kp3ds
    kp3d_processed[inds] = kp3d
ValueError: could not broadcast input array from shape (17,3) into shape (54,3)

Do you know what I need to do to avoid this error?

1. Also, does the cmu_panoptic have the 2d pose annotation for all the people appearing in every image?

I would appreciate it if you could help me with this, Thanks,

[Arthur151]

Hi，@mkhoshle, To prepare a dataloader for a dataset, please refer to https://github.com/Arthur151/ROMP/blob/f3c76725fff3f9cd0d3a90721315dcd5f23c3f58/romp/lib/dataset/MuPoTS.py#L48 Please note that, via function self.map_kps, we tranfer the different skeleton defination of each dataset (e.g. 17 joints in CMU P) to a unified format (our 54 joints). The mapping is achieved via: https://github.com/Arthur151/ROMP/blob/f3c76725fff3f9cd0d3a90721315dcd5f23c3f58/romp/lib/dataset/MuPoTS.py#L22

The skeleton of each dataset is defined in https://github.com/Arthur151/ROMP/blob/master/romp/lib/constants.py

You can define the skeleton of CMU P as CMUP_17, for example,

CMUP_17 = {'R_Ankle':0 ....}

Our 54 joints are defined as SMPL_ALL_54, https://github.com/Arthur151/ROMP/blob/f3c76725fff3f9cd0d3a90721315dcd5f23c3f58/romp/lib/constants.py#L45

Then the mapping would be self.kp3d_mapper = constants.joint_mapping(constants.CMUP_17, constants.SMPL_ALL_54)

[Arthur151]

I am not sure. Maybe this function https://github.com/Arthur151/ROMP/blob/f3c76725fff3f9cd0d3a90721315dcd5f23c3f58/romp/lib/visualization/visualization.py#L310 setting label_kp_order=True, would be helpful. Using to set up the dataset loader, you can use debug function at https://github.com/Arthur151/ROMP/blob/f3c76725fff3f9cd0d3a90721315dcd5f23c3f58/romp/lib/dataset/image_base.py#L700 to check the keypoints.

Yes, all multi-person datasets have this problem, more or less, including cmu_panoptic.