Adaptation to custom dataset

Hello, I’m trying to train the model on my custom dataset. I have created my synthetic dataset as similar as possible to the Linemod_preprocessed.

My steps:

• Created a folder in datasets copied from linemod folder.
• Modified the folder obj_kps putting the output generated by python3 gen_obj_info.py --help, the generated files only had center.txt, corners.txt, farthest.txt and radius.txt. It is not similar to the linemod dataset with farthest4.txt, farthest12.txt etc. - First question: How do I generate the other farthest missing files similar to linemod?
• I have changed the dataset_config/models_info.yml based on my measures obtained in the previous step
• I have modified the mydataset_dataset.py changing the width and height of my images, changing the references to linemod and pointing them to the config file of my new dataset
• Changed the preprocess_testset based on my needs
• in file common.py I have added my new dataset adding this part (doing copy and paste of linemod) :

        elif self.dataset_name == 'blm':
            self.n_objects = 1 + 1
            self.n_classes = 1 + 1
            self.blm_cls_lst = [
                1
            ]
            self.blm_obj_dict={
                'blm':1,
            }
            self.blm_id2obj_dict = dict(
                zip(self.blm_obj_dict.values(), self.blm_obj_dict.keys())
            )
            self.blm_root = os.path.abspath(
                os.path.join(self.exp_dir, 'datasets/blm/')
            )
            self.blm_kps_dir = os.path.abspath(
                os.path.join(self.exp_dir, 'datasets/blm/blm_obj_kps/')
            )
            self.val_test_pkl_p = os.path.join(
                self.exp_dir, 'datasets/blm/test_val_data.pkl',
            )
            prep_fd = os.path.join(
                self.blm_root, "preprocess_testset"
            )
            ensure_fd(prep_fd)
            self.preprocessed_testset_ptn = os.path.abspath(
                os.path.join(prep_fd, '{}_pp_vts.pkl')
            )
            self.preprocessed_testset_pth = self.preprocessed_testset_ptn.format(cls_type)
            self.use_preprocess = False

            blm_r_pth = os.path.join(self.blm_root, "dataset_config/models_info.yml")
            blm_r_file = open(os.path.join(blm_r_pth), "r")
            self.blm_r_lst = yaml.load(blm_r_file)

            self.val_nid_ptn = "/data/6D_Pose_Data/datasets/BLM/pose_nori_lists/{}_real_val.nori.list"

• I have added the intrinsic Matrix of my camera to the common.py.

Regarding my dataset, I have generated the images and labels using BlenderProc; I have the gt.yml, info.yml and everything except for the Masks. - Second question: Do I need to generate the masks or is it generated in some process inside? could that be the problem?

I have modified those necessary files here documented, but I still need to change some parts. This is the current output:

python3 -m train.train_blm_pvn3d --cls blm
/home/pytorch/PVN3D/pvn3d/common.py:173: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  self.blm_r_lst = yaml.load(blm_r_file)
/home/pytorch/PVN3D/pvn3d/common.py:134: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  self.lm_r_lst = yaml.load(lm_r_file)
/home/pytorch/PVN3D/pvn3d/common.py:173: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  self.blm_r_lst = yaml.load(blm_r_file)
cls_type:  blm
cls_id in blm_dataset.py 1
/home/pytorch/PVN3D/pvn3d/datasets/blm/blm_dataset.py:41: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  self.meta_lst = yaml.load(meta_file)
Train without rendered data from https://github.com/ethnhe/raster_triangle
Train without fuse data from https://github.com/ethnhe/raster_triangle
train_dataset_size:  10
cls_id in blm_dataset.py 1
val_dataset_size:  15
loading pretrained mdl.
/usr/local/lib/python3.7/dist-packages/torch/nn/_reduction.py:43: UserWarning: size_average and reduce args will be deprecated, please use reduction='mean' instead.
  warnings.warn(warning.format(ret))
{'bn_decay': 0.5,
 'bn_momentum': 0.9,
 'cal_metrics': False,
 'checkpoint': None,
 'cls': 'blm',
 'decay_step': 200000.0,
 'epochs': 1000,
 'eval_net': False,
 'lr': 0.01,
 'lr_decay': 0.5,
 'run_name': 'sem_seg_run_1',
 'test': False,
 'test_occ': False,
 'weight_decay': 0}
epochs:   0%|                                                                                                                                      | 0/25 [00:00<?, ?it/s]
train:   0%|                                                                                                                                     | 0/5000 [00:00<?, ?it/s]

How can I have more output and understand where it’s wrong?

[ethnhe]

Hi,

• For your first question, the default setting is to generate 8 keypoints for each object, which is the best setting in our experiments, and we think you do not need to get other numbers of keypoint like the LineMOD dataset, which we generate for ablation study in our paper. If you want to get them, you can modify the code here, by changing the num=8 to the number you want and also the name of the saving file.
• For your second question, yes, you need to provide the masks of target objects because our model needs to train the instance semantic segmentation branch. If you train your model for a single class of objects, you can follow the setting of LineMOD and set 1 as target objects and 0 as background in your mask. If you don’t know how to generate the mask of an object with BlenderProc, you can also try our raster_triangle to generate it with object ply model, gt object pose, and camera parameters.
• After these settings are finished, run the dataset provider scripts, mydataset_dataset.py in your case to visualize the data and check if all the scripts run correctly.

[ethnhe]

The keypoints are in m and are generated by the FPS algorithm with random initialization. The selected points may be different but that won’t affect the performance much. For your case, if the generated keypoints of your model are in m, you should check that your GT poses are in ‘cv’ coordinate system, rather than default poses from Blender.