Cannot reproduce results using COLMAP on data

Hello, I have been trying to do the following:

1. take a data, let’s say bmvs_stone
2. feed images to the COLMAP
3. generate the cameras_sphere.npz
4. train NeuS on these data

To generate the cameras_sphere.npz I used this code from the IDR project. This code has many times resulted in error because it was not able to find points for normalisation, here. After rerunning the COLMAP a few times, I got 2 points and it was able to generate the scale_mat_xx. However, the results after training are blurry, not good.

I have tried using automatic reconstruction in COLMAP and also this script from the LLFF project.

To convert the output from the COLMAP to camera_sphere.npz I used this:

import numpy as np
import os
import colmap_read_model as read_model
from scipy.spatial.transform import Rotation as R


DIR = "/home/uriel/Downloads/DTU/dtu_scan24"
DST = os.path.join(DIR, "cameras_sphere.npz")


def load_colmap_data(realdir):
    camerasfile = os.path.join(realdir, 'sparse/0/cameras.bin')
    camdata = read_model.read_cameras_binary(camerasfile)

    # cam = camdata[camdata.keys()[0]]
    list_of_keys = list(camdata.keys())
    cam = camdata[list_of_keys[0]]

    h, w, f, cx, cy = cam.height, cam.width, cam.params[0], cam.params[1], cam.params[2]
    # w, h, f = factor * w, factor * h, factor * f
    hwf = np.array([h, w, f]).reshape([3, 1])
    k = np.array([[f, 0, cx],
                  [0, f, cy],
                  [0, 0, 1]])

    imagesfile = os.path.join(realdir, 'sparse/0/images.bin')
    imdata = read_model.read_images_binary(imagesfile)

    w2c_mats = []
    bottom = np.array([0, 0, 0, 1.]).reshape([1, 4])

    names = [imdata[k].name for k in imdata]
    print('Images #', len(names))
    perm = np.argsort(names)
    for i in imdata:
        im = imdata[i]
        R = im.qvec2rotmat()
        t = im.tvec.reshape([3, 1])
        m = np.concatenate([np.concatenate([R, t], 1), bottom], 0)
        w2c_mats.append(m)

    w2c_mats = np.stack(w2c_mats, 0)
    c2w_mats = np.linalg.inv(w2c_mats)

    return k, c2w_mats, perm


k, rts, perm = load_colmap_data(DIR)
cameras = {}
for i in perm:
    r = rts[i, :3, :3]
    t = rts[i, :3, 3]
    wm = np.eye(4)
    wm[:3,:3] = k @ r
    wm[:3,3] = k @ t
    cameras['world_mat_%d' % i] = wm
    cameras['scale_mat_%d' % i] = np.eye(4)

np.savez(DST, **cameras)

The code is inspired by this script.

I think the problem is that COLMAP uses a different coordinate system then NeuS and IDR, which I suppose is the same as OpenGL, where COLMAP says: The local camera coordinate system of an image is defined in a way that the X axis points to the right, the Y axis to the bottom, and the Z axis to the front as seen from the image.

Do you have any ideas how to solve it? Thanks

[theFilipko]

I have figured this out. Here’s the working code:

import numpy as np
import os
import colmap_read_model as read_model


DIR = "/home/uriel/Downloads/stone"
DST = os.path.join(DIR, "cameras_sphere.npz")


camerasfile = os.path.join(DIR, 'sparse/0/cameras.bin')
camdata = read_model.read_cameras_binary(camerasfile)
list_of_keys = list(camdata.keys())
cam = camdata[list_of_keys[0]]

h, w, f, cx, cy = cam.height, cam.width, cam.params[0], cam.params[1], cam.params[2]
k = np.array([[f, 0, cx],
              [0, f, cy],
              [0, 0, 1]])

imagesfile = os.path.join(DIR, 'sparse/0/images.bin')
imdata = read_model.read_images_binary(imagesfile)
bottom = np.array([0, 0, 0, 1.]).reshape([1, 4])
names = [imdata[k].name for k in imdata]
perm = np.argsort(names)
cameras = {}
for i in perm:
    im = imdata[i+1]
    r = im.qvec2rotmat()
    t = im.tvec.reshape([3, 1])
    w2c = np.concatenate([np.concatenate([r, t], 1), bottom], 0)
    c2w = np.linalg.inv(w2c)

    r = c2w[:3, :3]
    r = r.T  # because of the load_K_Rt_from_P() function implemented in dataset.py
    # where the decomposed rotation matrix is transposed
    t = c2w[:3, 3]
    t = -t # -t because of the opencv projection
    # matrix decomposition function implementation
    # https://stackoverflow.com/questions/62686618/opencv-decompose-projection-matrix/69556782#69556782

    wm = np.eye(4)
    wm[:3,:3] = k @ r
    wm[:3,3] = k @ r @ t

    cameras['world_mat_%d' % i] = wm
    cameras['scale_mat_%d' % i] = np.eye(4)

np.savez(DST, **cameras)

[theFilipko]

@ahnaf1393 I recommend you to have a look at the COLMAP tutorial and use the GUI to play with settings of the Feature mapping, extraction and reconstruction. That is how you can have a better understanding of your data.