Installation issue: undefined symbol: __cudaPopCallConfiguration

On linux, when I try to install and use pytorch_scatter, I get undefined symbol: __cudaPopCallConfiguration immediately upon importing torch_scatter.

Using pytorch 1.0.0 and CUDA 9.0 is on the PATH (and include is on the CPATH):

$ python -c "import torch; print(torch.__version__)"
1.0.0
$ echo $CPATH
/usr/local/cuda-9.0/include
$ echo $PATH
/usr/local/cuda-9.0/bin:/afs/csail.mit.edu/u/d/davidbau/.conda/envs/p3t1/bin...

I’ve tried uninstalling and resintalling (without cache) on pip pip install --no-cache-dir torch_scatter, but the error remains. Any tips?

Details - ubuntu 16.04

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:        16.04
Codename:       xenial

Environment installed via conda using the following env.yml

name: p3t1
channels:
  - pytorch
dependencies:
  - python=3.6
  - cudatoolkit=9.0
  - cudnn=7.1.2
  - pytorch=1.0
  - torchvision
  - mkl-include
  - numpy
  - scipy
  - scikit-learn
  - matplotlib
  - graphviz
  - numba
  - jupyter
  - pyyaml
  - mkl
  - setuptools
  - cmake
  - cffi
  - ujson
  - tqdm
  - pip
  - pip:
    - torch-scatter

[davidbau]

Thanks for the tip! The system has multiple nvcc and even though the version on the PATH matched torch.version.cuda, it looks like setup.py was picking up the wrong one. It looks like the torch extension API is looking for CUDA_HOME. So doing this before pip install or python setup.py install fixes the problem (it’s not necessary for the right nvcc to show up on PATH or for the right include to be on CPATH - everything is keyed off of CUDA_HOME):

export CUDA_HOME=/usr/local/cuda-9.0

Another problem I was having while testing configurations was failing to sufficiently clean out binaries built with the wrong compiler. For others following along, I found this was enough to clean things:

pip uninstall torch-scatter
rm -rf build torch_scatter/*.so; python setup.py clean # within the torch_scatter sources

And then use --no-cache-dir when reinstalling with the proper environment variable.

export CUDA_HOME=/usr/local/cuda-9.0
pip install --no-cache-dir torch-scatter

Problem solved!

[rusty1s]

Does the PyTorch CUDA version torch.version.cuda match the one of the system?

Can you make sure the official PyTorch extensions run on your machine?