PyTorch-Geometric packages are incompatible with `conda-forge`

😵 Installation

There are 3 sets of PyTorch-Geometric packages available in the wild and old of them have issues:

1. conda-forge packages (https://anaconda.org/conda-forge/pytorch_geometric)

conda create -n tmp_pyg_cf -c conda-forge pytorch_geometric pytorch-gpu=1.10

The module fails to import:

$ python -c "import torch_geometric"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_cf/lib/python3.9/site-packages/torch_geometric/__init__.py", line 7, in <module>
    import torch_geometric.data
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_cf/lib/python3.9/site-packages/torch_geometric/data/__init__.py", line 1, in <module>
    from .data import Data
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_cf/lib/python3.9/site-packages/torch_geometric/data/data.py", line 3, in <module>
    from torch_geometric.typing import OptTensor
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_cf/lib/python3.9/site-packages/torch_geometric/typing.py", line 4, in <module>
    from torch_sparse import SparseTensor
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_cf/lib/python3.9/site-packages/torch_sparse/__init__.py", line 14, in <module>
    torch.ops.load_library(importlib.machinery.PathFinder().find_spec(
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_cf/lib/python3.9/site-packages/torch/_ops.py", line 110, in load_library
    ctypes.CDLL(path)
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_cf/lib/python3.9/ctypes/__init__.py", line 382, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /shared2/raimis/opt/miniconda/envs/tmp_pyg_cf/lib/python3.9/site-packages/torch_sparse/_convert_cuda.so: undefined symbol: _ZNK2at6Tensor6deviceEv

_ZNK2at6Tensor6deviceEv is at::Tensor::device() const, which is not a public symbol in PyTorch 1.10.

The packages have incorrect insufficient pinning of the PyTorch version:

$ conda search -i -c conda-forge pytorch_sparse
pytorch_sparse 0.6.10 py39hee7401c_1
------------------------------------
file name   : pytorch_sparse-0.6.10-py39hee7401c_1.tar.bz2
name        : pytorch_sparse
version     : 0.6.10
build       : py39hee7401c_1
build number: 1
size        : 901 KB
license     : MIT
subdir      : linux-64
url         : https://conda.anaconda.org/conda-forge/linux-64/pytorch_sparse-0.6.10-py39hee7401c_1.tar.bz2
md5         : ede306857f64cedff171f570d004b050
timestamp   : 2021-06-29 20:57:24 UTC
dependencies: 
  - __glibc >=2.17
  - cudatoolkit >=11.2,<12.0a0
  - libgcc-ng >=7.5.0
  - libstdcxx-ng >=7.5.0
  - pytest
  - pytest-cov
  - python >=3.9,<3.10.0a0
  - python_abi 3.9.* *_cp39
  - pytorch
  - pytorch-gpu >=1.8.0,<2.0a0
  - pytorch_scatter
  - scipy

P.S. It seems to work with PyTorch 1.8, but I specifically need PyTorch 1.10 with CUDA to use the CUDA Graphs (https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/).

1. pyg packages (https://pytorch-geometric.readthedocs.io/en/latest/notes/installation.html)

conda create -n tmp_pyg_pyg -c pyg -c conda-forge pyg pytorch-gpu=1.10

The module fails to import:

$ python -c "import torch_geometric"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_pyg/lib/python3.9/site-packages/torch_geometric/__init__.py", line 7, in <module>
    import torch_geometric.data
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_pyg/lib/python3.9/site-packages/torch_geometric/data/__init__.py", line 1, in <module>
    from .data import Data
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_pyg/lib/python3.9/site-packages/torch_geometric/data/data.py", line 3, in <module>
    from torch_geometric.typing import OptTensor, NodeType, EdgeType
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_pyg/lib/python3.9/site-packages/torch_geometric/typing.py", line 4, in <module>
    from torch_sparse import SparseTensor
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_pyg/lib/python3.9/site-packages/torch_sparse/__init__.py", line 19, in <module>
    torch.ops.load_library(spec.origin)
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_pyg/lib/python3.9/site-packages/torch/_ops.py", line 110, in load_library
    ctypes.CDLL(path)
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_pyg/lib/python3.9/ctypes/__init__.py", line 382, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /shared2/raimis/opt/miniconda/envs/tmp_pyg_pyg/lib/python3.9/site-packages/torch_sparse/_version_cuda.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKS

This is due to a mis-configured RPATH:

$ ldd /shared2/raimis/opt/miniconda/envs/tmp_pyg_pyg/lib/python3.9/site-packages/torch_sparse/_version_cuda.so
	linux-vdso.so.1 =>  (0x00007ffc159d5000)
	libc10.so => not found
	libtorch_cpu.so => not found
	libstdc++.so.6 => /shared2/raimis/opt/miniconda/envs/tmp_pyg_pyg/lib/python3.9/site-packages/torch_sparse/../../../libstdc++.so.6 (0x00007f7efb94a000)
	libgcc_s.so.1 => /shared2/raimis/opt/miniconda/envs/tmp_pyg_pyg/lib/python3.9/site-packages/torch_sparse/../../../libgcc_s.so.1 (0x00007f7efb934000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f7efb53e000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f7efb23c000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f7efb90c000)

Additionally, it needs GLIBC 2.27 (https://github.com/pyg-team/pytorch_geometric/issues/3477, https://github.com/Acellera/general/issues/164), but CentOS 7 has just 2.17.

1. PyPI packages (https://pytorch-geometric.readthedocs.io/en/latest/notes/installation.html)

conda create -n tmp_pyg_pip -c conda-forge python=3.9 pytorch-gpu=1.10 cudatoolkit=11.3
conda activate tmp_pyg_pip
pip install torch-scatter -f https://data.pyg.org/whl/torch-1.10.0+cu113.html
pip install torch-sparse -f https://data.pyg.org/whl/torch-1.10.0+cu113.html
pip install torch-cluster -f https://data.pyg.org/whl/torch-1.10.0+cu113.html
pip install torch-spline-conv -f https://data.pyg.org/whl/torch-1.10.0+cu113.html
pip install torch-geometric

The module fails to import:

$ python -c "import torch_geometric; print(torch_geometric)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_pip/lib/python3.9/site-packages/torch_geometric/__init__.py", line 7, in <module>
    import torch_geometric.data
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_pip/lib/python3.9/site-packages/torch_geometric/data/__init__.py", line 1, in <module>
    from .data import Data
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_pip/lib/python3.9/site-packages/torch_geometric/data/data.py", line 3, in <module>
    from torch_geometric.typing import OptTensor, NodeType, EdgeType
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_pip/lib/python3.9/site-packages/torch_geometric/typing.py", line 4, in <module>
    from torch_sparse import SparseTensor
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_pip/lib/python3.9/site-packages/torch_sparse/__init__.py", line 19, in <module>
    torch.ops.load_library(spec.origin)
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_pip/lib/python3.9/site-packages/torch/_ops.py", line 110, in load_library
    ctypes.CDLL(path)
  File "/shared2/raimis/opt/miniconda/envs/tmp_pyg_pip/lib/python3.9/ctypes/__init__.py", line 382, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libtorch_cuda_cpp.so: cannot open shared object file: No such file or directory

It trys to load an non-existing library:

$ find $CONDA_PREFIX -name libtorch_cuda_cpp.so

Maybe PyTorch has to be installed not form conda-forge.

Also, the package are build with the old C++ ABI (https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html):

nm -CD $CONDA_PREFIX/lib/python3.9/site-packages/torch_*/*.so | grep __cxx11

So, they are incompatible with the conda-forge ecosystem.

Environment

• PyG version (torch_geometric.__version__): 2.0.2
• PyTorch version: (torch.__version__): 1.10
• OS (e.g., Linux): CentOS 7.9
• Python version (e.g., 3.9): 3.9
• CUDA/cuDNN version: 11.3
• How you installed PyTorch and PyG (conda, pip, source): conda and pip
• Any other relevant information (e.g., version of torch-scatter):

[hadim]

The conda forge package seems to be working now (with the pytorch coming from conda-forge as well).

mamba create -n tmp_pyg_gpu -c conda-forge pytorch_geometric pytorch-gpu=1.10
mamba list -n tmp_pyg_gpu | grep 'pytorch\|cuda'
mamba run -n tmp_pyg_gpu python -c "import torch_geometric"

$ mamba list -n tmp_pyg_gpu | grep 'pytorch\|cuda'
cudatoolkit               11.1.1              ha002fc5_10    conda-forge
pytorch                   1.10.2          cuda111py39h930882a_1    conda-forge
pytorch-gpu               1.10.2          cuda111py39h788eb59_1    conda-forge
pytorch_geometric         2.0.4              pyhd8ed1ab_0    conda-forge
pytorch_scatter           2.0.8           cuda111py39h60b82b0_0    conda-forge
pytorch_sparse            0.6.10           py39h60b82b0_2    conda-forge

I also tested the below script:

import torch
from torch import Tensor
import torch.nn.functional as F

from torch_geometric.nn import GCNConv
from torch_geometric.datasets import Planetoid

DEVICE = "cuda"

dataset = Planetoid(root='.', name='Cora')

class GCN(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = GCNConv(in_channels, hidden_channels)
        self.conv2 = GCNConv(hidden_channels, out_channels)

    def forward(self, x: Tensor, edge_index: Tensor) -> Tensor:
        # x: Node feature matrix of shape [num_nodes, in_channels]
        # edge_index: Graph connectivity matrix of shape [2, num_edges]
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index)
        return x


model = GCN(dataset.num_features, 16, dataset.num_classes)
model = model.to(DEVICE)

data = dataset[0]
data = data.to(DEVICE)

optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

for epoch in range(2000):

    pred = model(data.x, data.edge_index)
    loss = F.cross_entropy(pred[data.train_mask], data.y[data.train_mask])

    # Backpropagation
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    print(epoch, loss)

and it works (CPU or GPU and on Linux only).

[raimis]

Yes, I can confirm the current packages form conda-forge are working. @hadim thanks for your effort. This can be closed.