Query with another index/ graph?

This is a question/ feature request.

My (cursory) understanding of the NNDescent algorithm makes me think it should be possible to increase query speed for a test set if we already had an index built on it. Is this the case?

If so, would this query be in scope for this project?

# a: NNDescent, b: NNDescent
a.query(b)

[lmcinnes]

I think this is possible, but tricky. The catch is that you want to to restrict the potential neighbors to be only those in the subset. This is hard to do and still end up with an adequte number of neighbors for each sample by simply subselecting from an existing index. I don’t actually see any easy ways to enable this.

[ivirshup]

In general, I think I’d be interested in contributing an implementation here. I’ve had some time to look over the paper (thanks for recommending that!), and had a couple questions – if you don’t mind. First, I’d like to make sure I’m on the same page about what I’d like to be able to do, and what’s described in the paper, then I had some questions about potential modifications.

The problem I would like to address is finding K neighbors from one dataset in another, essentially creating a bipartite graph. My understanding of this paper is that it only deals with building a joint graph containing all samples where all nodes have out-degree K. Is this right?

These main idea of using the predefined structure for faster search as in P-merge is definitely shared. I had a couple naive ideas for modification towards my ends: (1) Change the final step of P-merge. Instead of merging G- and H- into U, you could take the between dataset neighbors of U (i.e. 𝑈∩(𝐺⁺∪𝐻⁺)) replacing within dataset neighbors with sampled points, then optimize until convergence again. (2) Change the initialization so that you’ve got at least k samples from each dataset to be searched. Then remove all intra dataset edges and take top k remaining edges. Do you think these approaches might be worth pursuing?