n_jobs>1 returns an error for sparse input

I have a sparse matrix X, on which I can successfully run UMAP:

<100000x9630 sparse matrix of type '<class 'numpy.float64'>'
	with 266398 stored elements in List of Lists format>

In particular, nn = NNDescent(X, metric='cosine') works fine (it does raise a warning “Failed to correctly find n_neighbors for some samples”, but I’m ignoring it). However,

nn = NNDescent(X, metric='cosine', n_jobs=-1)

or any other non-default value of n_jobs returns an error here:

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Untyped global name 'tau_rand_int': cannot determine Numba type of <class 'numba.ir.UndefinedType'>

File "../../../anaconda3/lib/python3.7/site-packages/pynndescent/sparse_threaded.py", line 42:
def sparse_current_graph_map_jit(
    <source elided>
            for j in range(n_neighbors - np.sum(heap[0, i] >= 0.0)):
                idx = np.abs(tau_rand_int(rng_state_local)) % data.shape[0]

Versions:

UMAP 0.3.10
pynndescent 0.4.5
numba 0.46.0

[lmcinnes]

I believe the error is different since it was a new bug, introduced when I did some data structure refactoring and apparently missed a few spots.

In good news a lot of this should be made a lot more sane in version 0.5 where everything moves to numba parallelism controlled with numba’s new threading control features. This will mean that there will be only a single approach that covers both the serial and threaded cases rather than the current split of approaches that exists now. That should also fix some of the memory use issues with threaded use that you encountered earlier.

[dkobak]

Hi Leland, I just noticed that this seems to be resolved in 0.5 (at least everything works fine for me), and moreover the default behaviour (n_jobs=None) seems to be using n_jobs=-1 (btw, this was not entirely clear to me from the docstring). So I am closing this issue!