Easy entry point for DLC users

Is your feature request related to a problem? Please describe. Hi @lambdaloop! Thanks for this great package. We are hoping to have better integration with DLC users to DLC (see here: https://github.com/AlexEMG/DeepLabCut/blob/master/docs/Overviewof3D.md#more-than-2-camera-support)

One issue our users are having is an “easy entry” point. Right now, your docs are for setting up 2D then 3D tracking, but many users have the 2D done, and some images they want to calibrate. I really like the merger of DLC + anipose/calligator, so I want to help/ask to make this more seamless.

My understanding of the workflow for you is it requires users to use your file structure, which is a bit rigid. Would it be possible to make docs such that a user could jump in at this point: https://github.com/lambdaloop/anipose/blob/master/docs/github/start_3d.md#calibration-marker-configuration // https://anipose.readthedocs.io/en/latest/tutorial.html#calibrating-the-cameras

i.e. minimally, now after the 3d set up there is no link to your nice “readthedocs.io” demo (https://anipose.readthedocs.io/en/latest/tutorial.html#calibrating-the-cameras)

i.e. a more optimal solution would be: a simple Jupyter Notebook demo such that the person can navigate to their DLC project folder and then set up a simple config file for your system, then be able to run:

anipose calibrate
anipose triangulate
anipose label-3d

Also, for each of these functions, there is (at least I cannot find) any easy to read docstrings. For example, it is not clear what each function requires. (What file type, format, etc is points?)

triangulate(points, undistort=True, progress=False)
Given an CxNx2 array, this returns an Nx3 array of points, where N is the number of points and C is the number of cameras

Describe the solution you’d like Ideally, one could come in and use your functions (i.e. inhttps://github.com/lambdaloop/calligator, some below) with a clear entry point. i.e.

import anipose (or calligator directly?)

the use your functions: https://calligator.readthedocs.io/en/latest/api.html#module-calligator.boards anipose calibrate

then your triangulation functions:

anipose triangulate

triangulate(points, undistort=True, progress=False)
Given an CxNx2 array, this returns an Nx3 array of points, where N is the number of points and C is the number of cameras

triangulate_possible(points, undistort=True, min_cams=2, progress=False, threshold=0.5)
Given an CxNxPx2 array, this returns an Nx3 array of points by triangulating all possible points and picking the ones with best reprojection error where: C: number of cameras N: number of points P: number of possible options per point

triangulate_ransac(points, undistort=True, min_cams=2, progress=False)
Given an CxNx2 array, this returns an Nx3 array of points, where N is the number of points and C is the number of cameras

reprojection_error
Given an Nx3 array of 3D points and an CxNx2 array of 2D points, where N is the number of points and C is the number of cameras, this returns an CxNx2 array of errors. Optionally mean=True, this averages the errors and returns array of length N of errors

bundle_adjust_iter(p2ds, extra=None, n_iters=10, start_mu=15, end_mu=1, max_nfev=200, ftol=0.0001, n_samp_iter=100, n_samp_full=1000, error_threshold=0.3, verbose=False)
Given an CxNx2 array of 2D points, where N is the number of points and C is the number of cameras, this performs iterative bundle adjustsment to fine-tune the parameters of the cameras. That is, it performs bundle adjustment multiple times, adjusting the weights given to points to reduce the influence of outliers. This is inspired by the algorithm for Fast Global Registration by Zhou, Park, and Koltun

bundle_adjust(p2ds, extra=None, loss='linear', threshold=50, ftol=0.0001, max_nfev=1000, weights=None, start_params=None, verbose=True)
Given an CxNx2 array of 2D points, where N is the number of points and C is the number of cameras, this performs bundle adjustsment to fine-tune the parameters of the cameras

Describe alternatives you’ve considered We have considered making DLC n-camera support, but we would rather not 😃

Additional context Right now the documentation is making this hard for us to guide users

Cheers, DLC hackathon subgroup “3d, yeah!”

[lambdaloop]

@SjoerdBruijn I don’t know Simi, but most tools (including Anipose) use the same camera model, so it’s doable to translate across them. The calibration file for Anipose is a toml file, which you could write to based on your Simi parameters. Here’s an example file:

[cam_0]
name = "54138969"
size = [ 1000, 1000,]
rotation = [ 0.4673375748186822, 2.173592643964762, -1.778427743746773,]
translation = [ -219.3059666108619, 544.4787497640639, 5518.740477016156,]
distortions = [ -0.207098910824901, 0.247775183068982, -0.00142447157470321, -0.000975698859470499, -0.00307515035078854,]
matrix = [ [ 1145.04940458804, 0.0, 512.541504956548,], [ 0.0, 1143.78109572365, 515.4514869776,], [ 0.0, 0.0, 1.0,],]

[cam_1]
name = "55011271"
size = [ 1000, 1000,]
rotation = [ 1.75656787919027, 0.3622445290737643, -0.2740327492410326,]
translation = [ 103.9028206775199, 395.6716946895197, 5767.97265758172,]
distortions = [ -0.194213629607385, 0.240408539138292, -0.0027408943961907, -0.001619026613787, 0.00681997559022603,]
matrix = [ [ 1149.67569986785, 0.0, 508.848621645943,], [ 0.0, 1147.59161666764, 508.064917088557,], [ 0.0, 0.0, 1.0,],]

Similarly, if you use calligator (now renamed to anipose-lib), you can create a CameraGroup object and update the parameters by calling the appropriate functions.

When converting calibration parameters across systems in the past, the main issue that I found has been the rotation and translation parameters, which may be in a different format. Anipose uses the OpenCV format for calibration parameters, with rotation specified as an axis rotation vector (can be converted to/from matrix using the Rodrigues function).

[lambdaloop]

Sounds good! It sounds like the most pressing missing thing is a Calligator tutorial then. We’ll put that together in the coming weeks!