LLVM-Error when using mahalanobis metric with larger datasets

Hello!

First: Thanks for this splendid project! It’s looking great, especially the semi-supervised part.

When I try to use Mahalanobis-distances with larger datasets, so the < 4000 codepath is not taken, I get an LLVM IR parsing failure. The other codepath works just fine.

Steps to reproduce:

Version: umap-learn-0.3.2

import numpy as np
from umap import UMAP

matrix = np.random.rand(5000,50)
umap = UMAP(n_components=2, n_neighbors=30, metric='mahalanobis', metric_kwds={'V': np.eye(50)})

umap_model = umap.fit_transform(matrix)

Resulting in

Failed at nopython (nopython mode backend)
LLVM IR parsing error
<string>:1121:137: error: invalid use of function-local name
  %".786" = extractvalue [1 x {i8*, i8*, i64, i64, double*, [2 x i64], [2 x i64]}] [{i8*, i8*, i64, i64, double*, [2 x i64], [2 x i64]} %".785"], 0

Full stracktrace (redacted for brevity) here: https://gist.github.com/johanbev/1918108e65f014600b2e44affcc35fee

[johanbev]

Thanks for the quick reply! It seems that this issue is caused by jitting the make_nn_descent() in nopython mode. My guess is that njit()-variant cannot access the outer *dist_args parameter somehow. Changing this into @jit and turning off parallelization fixed the issue. Same issue and solution with seuclidean.

Very cursory benchmarking with euclidean distance suggests that there is no performance impact with running nn_descent in object-mode.

[lmcinnes]

Thanks for the report – I’m not sure quite what the cause is here, but I’ll try to track down what might be at issue soon. I’m working on other projects right now, so it might be a little while before I can get to this in earnest. Sorry.

On Fri, Sep 14, 2018 at 6:15 AM Johan [e-mail] wrote:

Hello!

First: Thanks for this splendid project! It’s looking great, especially the semi-supervised part.

When I try to use Mahalanobis-distances with larger datasets, so the < 4000 codepath is not taken, I get an LLVM IR parsing failure. The other codepath works just fine.

Steps to reproduce:

Version: umap-learn-0.3.2

import numpy as np from umap import UMAP

matrix = np.random.rand(5000,50) umap = UMAP(n_components=2, n_neighbors=30, metric=‘mahalanobis’, metric_kwds={‘V’: np.eye(50)})

umap_model = umap.fit_transform(matrix)

Resulting in

Failed at nopython (nopython mode backend) LLVM IR parsing error <string>:1121:137: error: invalid use of function-local name %“.786” = extractvalue [1 x {i8*, i8*, i64, i64, double*, [2 x i64], [2 x i64]}] [{i8*, i8*, i64, i64, double*, [2 x i64], [2 x i64]} %“.785”], 0

Full stracktrace (redacted for brevity) here: https://gist.github.com/johanbev/1918108e65f014600b2e44affcc35fee

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/lmcinnes/umap/issues/142, or mute the thread https://github.com/notifications/unsubscribe-auth/ALaKBYY3yO0qcg9PtOokas_xrnhqhBCMks5ua4HHgaJpZM4WpAR5 .