Question related to Input Preparation

Hello, thank you for your sharing, I really appreciate your KPConv work and I am working on implement it in PyTorch. But I was confused when I try to understand the code of data preparation part because there are so many encapsulations or wrappers. So would you please give me some high level idea about the implementation. Basically my questions are

1. About the flat_inputs variable : In my understanding, you build the flat_inputs which consists of the points and its neighbors for each layer before session run. But how can you calculate the neighbors of each layer before the specific point cloud is sent into the network?
2. For the convolutional part, I already know how to build the matrix and calculate the output feature for the batch input, but before conv I need know the neighborhood indices of each point, I have no idea how to calculate the neighbor for each point cloud in parallel, obviously if I try to calculate the neighbor indices one by one and then concatenation it will be very time comsuing. So in your implementation, how do you solve this problem ?

[HuguesTHOMAS]

Hello @XuyangBai,

Thank you for your interest, a PyTorch implementation is a great idea, unfortunately I did not have any time to do it, but I can help you for sure.

The funny thing is that your second question is the answer to your first question and your first question is… Well, the answer of your second question. Here is my line of thoughts when I implemented the whole thing.

First, as you noticed, we need the matrix of neighbors in the convolution operator, and finding neighbors in a point cloud is not a task that you can easily implement on a GPU. In fact it is quite well optimized for CPU, with the used of KD-Tree to accelerate the search time. Therefore, I decided to pre-compute the neighbors, before sending them to the GPU. It is the same for pooling indices, which are the point neighbors from one layer to the other. We thus need the points of each layer beforehand, which is possible because our pooling operation is purely geometric (we do not use the features of the network to adapt the pooling location).

In brief, there are two groups of operations:

• The input pipeline: generate the input point clouds, process them with data augmentation, subsample for every layer, find neighbors/pooling indices. Everything here is done on CPU, with an parallel input queue (8 threads precomputing a queue of input batches).

• The network graph: all the layer ops in GPU. The network takes the first batch from the input queue , process it, etc.

Tensorflow already have a Dataset class that implements parallel input queues with multiple threads, but it need every operation to be a tensorflow op. This is why I implemented 2 tensorflow C++ wrappers in the folder tf_custom_op, one for neighbor search (with the nanoflann library), and one for subsampling.

The python C++ wrappers in the folder cpp_wrappers are not used by the network or the input pipeline, they only get called at preprocessing steps (the part of the code that run only once the first time your start a training). Actually there is only one subsampling wrapper, and it can be replaced by a python function if you wish.

The flat_inputs variable is a list of all the input tensors, returned by the input pipeline. It contains a lot of things, with some variables that are not really necessary, but that I used for debugging. For convenience, I convert it to a dictionary afterwards in the model class.

I hope my explanations are clear enough. If you have any other question, feel free to ask. Did you catch the trick to manage the shadow neighbors? And the bacth stacking process, with the batch length calibration? Another part of the code that is not explained in the paper is the neighborhood calibration function, which is quite important for speed.

Best, Hugues

[XuyangBai]

Hi all, I have just finished a initial version of PyTorch implementation here.