Problem with depth-image

Hello, I try to train DenseFusion on my own dataset. But I seem to have a problem with the depth-image and the pointcloud which is generated from it by

cam_scale = 1.0
pt2 = depth_masked / cam_scale
pt0 = (ymap_masked - self.cam_cx) * pt2 / self.cam_fx
pt1 = (xmap_masked - self.cam_cy) * pt2 / self.cam_fy
cloud = np.concatenate((pt0, pt1, pt2), axis=1)
cloud = cloud / 1000.0

My depth image is in mm 16 bit (synthesized with NDDS). A projection of the point-cloud into the color-image: As you can see the objectshape seems to be correct, but the pose is wrong. I checked the corresponding depth image, there the pose is correct.

I also have a question about the values in the array from depth-image. In the code we take the depth-image as an array with depth = np.array(Image.open(self.list_depth[index])) . If I print the values of my own depth-image it starts with 1100, which should be the depth in mm. But if I print the same values for a depth-image from linemod the array starts with 0, which would mean a depth of 0 mm. Can you tell me why?

[KatharinaSchmidt]

I didn’t use a specific .mat file. In my dataset.py I just read the informations out of the .json file:

# open the annotation-file
with open(self.list_json[index]) as jsonfile:
            object_data = json.load(jsonfile)

# get the position from annotations
target_t = np.array(object_data['objects'][0]['location']) * 10 # NDDS gives units in centimeters

# get the rotation from annotation file
rot= np.array(object_data['objects'][0]['pose_transform'])[0:3,0:3]

# get the corners of bounding boxes from annotations
bbox = object_data['objects'][0]['bounding_box']
rmax, cmax = np.asarray(bbox['top_left'])
rmin, cmin = np.asarray(bbox['bottom_right'])
rmax, rmin, cmax, cmin = int(rmin), int(rmax), int(cmin), int(cmax)

with this code you get the pose in correct units to train densefusion with linemod-code.

[an99990]

if depth map is in meters, why are they dividing by 1000 ? @huckl3b3rry87 @akeaveny @KatharinaSchmidt . My mesh are in meters so i removed the /1000, should i bring it back ?

        cloud = cloud / 1000.0

        model_points = self.pt[obj] / 1000.0
        dellist = [j for j in range(0, len(model_points))]
        dellist = random.sample(dellist, len(model_points) - self.num_pt_mesh_small)
        model_points = np.delete(model_points, dellist, axis=0)

        target_r = np.resize(np.array(meta['cam_R_m2c']), (3, 3))
        target_t = np.array(meta['cam_t_m2c'])
        add_t = np.array([random.uniform(-self.noise_trans, self.noise_trans) for i in range(3)])

        if self.add_noise:
            cloud = np.add(cloud, add_t)

        target = np.dot(model_points, target_r.T)
        if self.add_noise:
            target = np.add(target, target_t / 1000.0 + add_t)
            out_t = target_t / 1000.0 + add_t
        else:
            target = np.add(target, target_t / 1000.0)
            out_t = target_t / 1000.0