questions about the loss function

Hi, Clément Pinard It’s an impressive job that you have done in this project and the code is written beautifully. May I ask you some questions about the loss function? For the photometric_reconstruction_loss, it use the predicted depth and pose to get the grid from ref_imgs to tgt_img. Thus, we can use the F.grid_sample to generate tgt_img from ref_imgs. But in theory, we can also generate ref_imgs from tgt_img. Do you have any idea about how to realize this operation?

[huagl]

I see. I found the problems that you told when I tried to implement my naive idea. Thanks for your patience~

[ClementPinard]

If you do that you won’t have any parallax effects : for instance in a lateral movement, close points will move faster on the screen than far points. If you use a constant ref_depth they will all move in the image at the same speed, as if you were looking at a textured plane instead of the real world.

I agree that the need for 2 depth map is counter-intuitive, you feel that with 1 depth map and 1 pose vector, you have all the information to compute the new depth map, but this is only true for direct warping. The problem is that direct warping is much more difficult to implement as a differentiable than inverse warping.

Inverse warping needs the depth map corresponding the new pose view point, and the thing you are trying to warp (be it image or depth) corresponding to the original pose viewpoint. That way, contrary to direct warp, each point of the warped image/depth has a coordinate in the source image, and gradient of pixel color with respect to coordinate is very easy to compute. I wrote a piece on that in my thesis (https://pastel.archives-ouvertes.fr/tel-02285215/document chapter 4)

If you are searching for direct warp based optimization, you can have a look at Pytorch3D. They implement a method for 2D image optimization based on direct rendering of 3D primitives. You just need to convert your depth map to 3D mesh (should be easy) and plug their solution.