-1 entries in neighbor_graph

I get some -1 entries in the kNN graph when I build it on a sparse matrix with cosine distance. Is this intentional? Does -1 mean X.shape[0]-1? When I run query(X), I don’t get any negative elements. E.g.:

X = scipy.sparse.csr_matrix(np.random.randn(10000,1247)>3)
nn = pynndescent.NNDescent(X, metric='cosine', n_neighbors=15)

Now nn.neighbor_graph[0] has some values equal to -1 but nn.query(X, k=15)[0] does not.

Update: Forgot to say that I do get a warning “UserWarning: Failed to correctly find n_neighbors for some samples.Results may be less than ideal. Try re-running withdifferent parameters.” from NNDescent. Maybe that’s what -1 indicate? But then query() does not return any negative elements and does not complain. How should one approach this?

[dkobak]

For your information, I’ve been looking at this dataset: https://johnhw.github.io/umap_primes/index.md.html. Turns out, this (all neighbors -1) happens for around 10% of points (at least out of the first 100K numbers). It seems these are mostly prime numbers and UMAP mostly keeps them where they were at initialization.

[camiel-m]

Hi Leland, I ran into a similar issue with -1’s in the neigbor_graph.

When I run the snippet below I see that there are no -1’s in the graph.

X = scipy.sparse.csr_matrix(np.random.randn(10000,1247)>3)

nn = NNDescent(data=X, metric="cosine", n_neighbors=10)
indices, distances = nn.neighbor_graph
np.sum(nn.neighbor_graph[0]==-1)

output: 0

However, If I query the graph there are suddenly -1’s.

indices2, distances2 = nn.query(X)
np.sum(nn.neighbor_graph[0]==-1)

output: 62251

This seems odd since the underlying graph shouldn’t be affected by querying (new) data. The number of -1’s varies when using different metrics, but there’s always some.

I’m on pynndescent 0.48.1 and numba 0.49.1.