faiss-gpu: index_cpu_to_gpu() hangs (doesn't occur with the conda package)

When I install faiss-gpu via pip, index_cpu_to_gpu() seems to hang forever. For example, this code sample hangs for me:

import faiss

index_flat = faiss.IndexFlatL2(16)
gpu_index_flat = faiss.index_cpu_to_gpu(faiss.StandardGpuResources(), 0, index_flat)

The index_cpu_to_gpu() will hang, spinning a CPU at 100% seemingly forever, mostly in libnvidia-ptxjitcompiler.so.510.54.

I confirmed this on two different machines with different GPUs (A100 and A10G).

I found two workarounds:

1. Downgrade to faiss==1.5.3. I hit the problem with 1.6.0 or later.
2. Use the conda package. I don’t observe the issue with conda, even with the latest version of faiss-gpu (1.7.2)

Maybe faiss-gpu wheel isn’t built with CUDA 11 support, and that’s why it doesn’t work with A100 / A10G?

[anubhav562]

Hey everyone!

Thanks a lot for this chain! My issue got solved!

Issue: I was not able to use FAISS on the NVIDIA A100. The FAISS index was not getting pushed to the GPU!

For people who want a solution at one place. Please follow these steps below:

• As faiss-gpu 1.7.3 is not available on pip. You can download the artifact directory (a directory containing multiple wheel files based on different systems) from here -> https://github.com/kyamagu/faiss-wheels/actions/runs/3487300515 . Go to bottom of the page -> locate artifacts and download it to your machine.

• Once you have the artifacts folder, you need to see which file is the most suitable for you system. For example: I used the following file: faiss_gpu-1.7.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl . The filename itself indicates the configurations of a system. In the filename cp38 could mean CPython 3.8 (that is what I think). The artifacts directory also has many wheels for different systems. Locate the one compatible for your machine.

• Once you have located your compatible wheel just run the following: python -m pip install /path/to/wheel/faiss_gpu-1.7.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

And you are good to go!

Thanks to @kyamagu for maintaining the repo!

[igor0]

OK, yeah, that seems to work! (Not sure how that’s different from what I was trying previously.)

That wheel seems to fix my trivial repro, so it addresses the issue, at least as far as I can tell. Presumably this particular wheel won’t work for someone who has CUDA 10, but that’s a different problem. That’s why PyTorch versions are so complicated.